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EXECUTIVE SUMMARY 
Background 


In 2015, Bossier Parish Community College (BPCC), located near Shreveport in northwest Louisiana, 
received a four-year grant through the United States Department of Education’s (US DOE) First in the 
World (FITW) grant program. 


As set forth in its FITW grant application, BPCC’s initiative consists of two interrelated strategies: 


1) — strengthen student learning through gamification of a native, untethered, mobile learning 
application for developmental education courses, and 


2) integrate the learning application into an integrated data platform (IDP) of campus data 
streams, creating a predictive flow model illuminating patterns of student persistence and 
learning. 

BPCC’s approach of creating mobile applications for one English (English 99) and two math (Math 98 
and Math 99) developmental education courses builds on its prior efforts designing Open Campus™,' a 
free online platform that provides access to developmental education course content to students 
anywhere in the world. 


BPCC’s goal is to improve the academic performance of students enrolling in developmental education 
courses, increase these students’ rates of persistence and degree attainment, and develop methods 
rooted in data science for predicting the success of developmental education students and intervening 
with students predicted to have low odds of success. 


This Evaluation 


All FITW grants require an evaluation to be conducted by an independent third-party evaluator. 
BPCC’s evaluation is being conducted by Giani Consulting & Evaluation, LLC. The evaluation consists 
of 


e animpact evaluation using a randomized controlled trial (RCT) design to produce unbiased 
estimates of the effect of the intervention on student outcomes, and 

e animplementation evaluation that seeks to explore the extent to which BPCC implemented 
with fidelity the activities included in its grant application—as well as the factors that were most 
influential in facilitating or obstructing implementation. 


The impact evaluation focuses exclusively on the impact of the mobile apps on student performance, 
whereas the implementation evaluation examines the implementation of both the mobile apps and the 
integrated data platform. 


For the impact evaluation, BPCC provided the evaluator with data on all courses that students in the 
analytic sample attempted prior to and through the Spring 2019 semester, including course credit and 
grade information, credentials earned by students through that semester, and data on students’ 
background characteristics. This data was used to assess the impact of access to the mobile apps on 
students’ course performance - as indicated by course pass rates, grades in the developmental education 
courses, semester GPA, and persistence into the next semester — and the attainment of credentials. 


' For more on BPCC’s Open Campus™, see http://www.bpcc.edu/opencampus. 
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The evaluator collected data for the implementation evaluation through a variety of methods, including 


e annual site visits to interview students, faculty, and staff at BPCC to understand how 
implementation was proceeding, 
e surveys administered to various stakeholders to gauge responses to the intervention, and 
e regular phone calls with FITW staff at BPCC to assess progress toward goals and activities 
outlined in BPCC’s logic model for this project (see Table 1: Logic Model for BPCC's FITW 
Initiative). 
This final Annual Performance Report is designed to 


e document the fidelity of implementation and assess factors influencing implementation through 
the final year of the grant period (2018-2019), 

e summarize the results of the impact evaluation for both the first (Spring 2017) and second (Fall 
2017) cohorts, 

e estimate the impact of access to the mobile apps on students’ credential attainment rates 
through Spring 2019. 


The rest of this executive summary overviews key findings from this final annual performance report. 
Summary of Year 4 Findings 


1. A Beta Version of the Mobile Apps was Piloted with Two Cohorts, and a Fully 
Functioning Version of the Mobile Apps was Released in 2018-19. 


BPCC piloted the Phase 1 mobile apps with two cohorts of students. Cohort 1 recetved access to the 
apps in Spring 2017, and Cohort 2 received access in Fall 2017. Both cohorts used a beta version of the 
apps that are more accurately described as mobile-responsive versions of the Open Campus website 
that students were instructed to save as an icon on the home screen of their smartphone, rather than 
native mobile apps. Both cohorts were randomly assigned to receive access to the mobile apps or not 
by randomly assigning course sections to treatment and control conditions. 


In the final year of the grant, BPCC released fully functioning versions of the mobile apps that are 
available for download through the Apple App Store and Google Play. The apps now provide a more 
seamless user experience and additional functionality including auto-grading of quizzes and 
assignments, a gradebook, better integration of analytics functionality, greater accessibility (that is, 
compliance with the Americans with Disabilities Act), enhanced administrator/faculty tracking of 
student engagement, and compatibility with 1OS and Android, among other updates. 


2. The Integrated Data Platform (IDP) Has Seen Widespread Use Among BPCC 
Advisors, but Less Use Among Faculty. 
BPCC entered the third year of implementation of its integrated data platform, Civitas Learning’s* 


Inspire platform. This tool enables faculty and front-line staff to access in a single location various data 
related to students’ backgrounds, target intervention and outreach efforts toward students identified as 


? Civitas Learning, Inc., provides resources to higher education that help bring together diverse sources of data; for 
more information, see https://www.civitaslearning.com/about. 
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being at-risk of dropping out, and document these interventions so that other stakeholders are aware of 
the efforts that have been taken to support students. 


During 2017-2018, more than 150 BPCC staff received training on Inspire through a total of 16 
workshops. The staff served constitute more than 75 percent of the BPCC personnel who advise 
students. Inspire was used to make more than 10,000 outreaches to students during that academic year. 
The IDP has been integrated into the workflow of BPCC advisors, but a minority of faculty are 
regularly using the platform. 


3. Faculty Remain Engaged with the FITW Initiative, but Contextual Factors 
Impede Implementation. 


The majority of instructors interviewed supported the developmental education mobile apps and 
acknowledged the benefits of the IDP. However, contextual factors impede implementation. Two 
factors in particular have posed the greatest challenge. 


First, instructors felt that limitations of the Phase 1 mobile apps hindered their utility. For example, the 
mobile apps were not available through common mobile app repositories such as the App Store and 
Google Play, and instructors could not log into the apps and see how their students were using them. 
The Phase 2 mobile apps have addressed many of the issues raised by faculty, but incorporation of the 
mobile apps into course expectations, assignments, and instruction is uneven. However, the vast 
majority of instructors support the mobile apps and recommend them to students, even if instructors 
do not integrate the apps into their courses. 


Second, instructors’ perceptions of the IDP were colored by their dissatisfaction with advising students. 
Although student advising is a required part of faculty work per BPCC faculty contracts, the majority of 
instructors felt they were not sufficiently compensated or trained for this work. Faculty buy-in for the 
IDP will likely be heavily influenced by whether they perceive the tool as an added burden or a way to 
simplify and facilitate the student advising they are required to do. 


4. Student Engagement with the Mobile Apps was Modest, but Reasonable. 


Roughly 20 percent of students assigned to the treatment group for both Cohorts engaged with the 
mobile apps. Cohort 2’s engagement with the apps was somewhat deeper, as students who used the 
apps watched more videos and completed a higher percentage of quizzes compared to students in 
Cohort 1. Nevertheless, many at BPCC were discouraged at the low overall rate of engagement among 
students. 


Although the lack of a common pre-test for all students in the analytic sample prevented an analysis of 
how students’ prior achievement influenced their engagement with the apps, anecdotal evidence 
suggests students who were performing well in their developmental education courses and students 
who were extremely disengaged from their courses both had low rates of using the mobile apps. In 
contrast, students who were engaged with the course but struggling academically were more likely to 
use the apps. Given that the mobile apps were used as a supplemental instructional resource in many 
classrooms, it is reasonable that students who needed additional support would be mote likely to take 
advantage of the resource compared to students who were already performing well. 
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5. Despite Low Student Engagement, the Impact Evaluation Results Are 
Promising. 


Although only about one in five students in the treatment group used the mobile apps, the results of 
the impact evaluation demonstrate that students who used the apps performed significantly better than 
students in the control group across a variety of outcomes. The results of the impact evaluation for 
both cohorts is summarized below. 


Developmental Education Course Grades 


Students given access to the mobile apps outperformed students who were not in their developmental 
education courses. The smallest benefit was found in Engl 99 of 0.07 grade points and the largest was 
in Math 98 of 0.19 grade points for an average difference of 0.14 grade points across courses. The 
statistical analysis produced an equivalent estimate of the average treatment effect of 0.136 grade 
points, a statistically significant difference (p = 0.032). 


2.00 1.77 1.84 1.77 1.82 1.81 
1.80 1,70 +67 
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1.40 

120 
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Figure 1: Developmental Education Course Grades for Both Cohorts, by Treatment Status and Course Number 
Developmental Education Course Passing 


This improvement in course performance translated into a higher likelihood of passing developmental 
education courses for students assigned to the treatment group. Across both cohorts and all three 
course numbers, students in the treatment group were more likely to pass their developmental 
education courses compared to students in the control group. This benefit ranged from a low of 2.6 
percentage points in Engl 99 to 9.7 percentage points in Math 99, for an overall benefit of 5.3 
percentage points across courses. The estimate from the statistical model of students in the treatment 
group being 5.5 percentage points more likely to pass their developmental education course was 
statistically significant (p = 0.009). 
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Figure 2: Developmental Education Course Pass Rates for Both Cohorts, by Treatment Status and Course Number 


This positive effect of access to the mobile apps for students in Math 99 was also evident in the Cohort 
2 students’ Fall 2017 GPA. Math 99 students in the treatment group earned a semester GPA 0.28 
points higher than did control-group students, and the statistically estimated treatment effect for Math 
99 students was 0.30 (p = 0.026), a statistically significant difference. There was limited effect of the 
mobile apps on the Cohort 2 GPA for the other two developmental education courses—Math 98 and 
English 99—and the effect on Fall 2017 GPA for the combined sample was not statistically significant 
(p = 0.275). 


Next Semester Persistence 


Surprisingly, while the effects of the mobile apps on student performance in developmental education 
courses was larger for students in the two math courses, the largest effect on persistence was for 
students in English 99. The average difference in rates of persistence to the next semester was 3.3 
percentage points across all three courses, with treatment students in English 99 demonstrating a 9.3 
percent higher persistence rate compared to that of control students. This difference for English 99 
students was estimated to be significant (p = 0.041) in the statistical models of next-semester 
persistence. However, the estimate of the average treatment effect across courses of 2.5 percentage 
points was not found to be statistically significant (p = 0.247). 
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Figure 3: Next Semester Persistence Rates for Both Cohorts, by Treatment Status and Course Number 


Next Semester GPA 


Perhaps most impressively, in the semester after the intervention, students in the treatment group 
earned higher GPAs than did students in the control group. Across all three courses, treatment students 
earned a GPA 0.11 points higher than did control students, and the estimated treatment effect of 0.14 
was found to be marginally statistically significant (p = 0.053). This effect was driven largely by students 
in Math 99. The treatment effect for students in this course was 0.37, which was also a statistically 
significant difference (p = 0.001). 
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Figure 4: Next Semester GPA for Both Cohorts, by Treatment Status and Course Number 
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Next Semester Course Passing Rates 


Figure 4 shows the percentage of attempted courses that students passed in the semester following the 
intervention semester. Overall, the treatment group passed 69.1% of the courses they attempted, 
compated to 66.2% for the control group, a difference of 2.9 percentage points. The estimate of the 
treatment effect was 3.7 percentage points, a marginally significant difference (p = 0.074). The largest 
difference was found in Math 99, where students in the treatment group passed 7.7% more of their 
courses compared to students in the control group. The statistical estimate of the average treatment 
effect for students in Math 99 of 8.2 percentage points was statistically significant (p = 0.016). 
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Figure 5: Next Semester Course Passing Rates for Both Cohorts, by Treatment Status and Course Number 


Next Semester Credits Earned 


We also find students given access to the mobile apps earned marginally more credits in the following 
semester compared to students in the control group. Students in the treatment group earned 5.62 
credits on average in the subsequent semester compared to students in the control group who earned 
5.15 credits, a difference of 0.47 credits. This is similar to the statistical estimate of the average 
treatment effect of 0.46 credits, which was a marginally significant difference (p = 0.072). 
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Figure 6: Next Semester Credits Earned for Both Cohorts, by Treatment Status and Course Number 


Attainment Analyses 


The final outcome investigated in this impact evaluation is the attainment of credentials through Spring 
2019. It should be mentioned that the timeframe for evaluating credential attainment is relatively 
limited. Students in Cohort 2 were enrolled in developmental education courses in Fall 2017, giving 
them only two years to have earned a credential by Spring 2019 after having begun in developmental 
education courses. The literature is clear that few students who begin in developmental education earn 
a credential within two years. Nevertheless, we investigated this outcome to explore whether begin 
given access to the mobile apps accelerated students’ persistence through college sufficiently to produce 
effects on short-term credential attainment. 


Figure 7 displays the rates of credential attainment for the treatment and control groups and across 
course numbers. Overall, students given access to the mobile apps were 0.5% more likely to complete a 
credential (certificate or associate’s degree) compared to students in the control group, but this 
difference was not found to be statistically significant (p = 0.618). Analyses investigating the effect 
separately on associate’s degree attainment and certificate attainment also found no significant 
differences between the treatment and control groups, nor were there significant benefits on credential 
attainment for any of the three course numbers. However, the statistical models estimated that students 
in the treatment group in Math 99 were 1.7 percentage points more likely to earn any credential 
compared to students in the control group. Although this difference was not found to be statistically 
significant, it is a non-trivial improvement in the credential attainment rate. Given that the Math 99 
control group’s baseline credential attainment rate was 20.1%, a 1.7 percentage point improvement 
equates to an 8% increase in the credential attainment rate for this subgroup. If these cohorts are 
followed into the future, this may prove to be a statistically significant difference as well as a practically 
important one. 
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Figure 7: Credential Attainment Rates for Both Cohorts, by Treatment Status and Course Number 


Summary of Results of Impact Evaluation 


There was considerable variation by course number and semester in the effects of the mobile apps on 
students’ performance in the intervention semester and in the future. Nevertheless, the results of the 
impact evaluation suggest not only that the mobile apps significantly improved students’ performance 
in their developmental education courses, but also that the apps affected students’ likelihood of success 
in courses the following semester. Although there was insufficient statistical evidence to conclude that 
being given access to the mobile apps improved students’ credential attainment rates, students in the 
treatment group did complete credentials at higher rates than students in the control group across all 
course numbers. Following these cohorts further in the future may reveal that the effect on attainment 
is in fact statistically significant as well as practically important. 


Overall, despite being a relatively modest intervention in terms of the resources required to sustain it, 
the Open Campus mobile apps developed by BPCC significantly improved many important student 
outcomes. Sustaining and scaling this intervention may allow BPCC to continue to move the needle on 
the success rates of developmental education students, and the results provide rigorous evidence that a 
well-designed technological intervention can measurably improve college student performance. 
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INTRODUCTION 


In August 2015, Bossier Parish Community College (BPCC) was awarded a First in the World (FITW) 
grant from the United States Department of Education (US DOE). 


BPCC proposed to fundamentally transform how developmental education content is provided to 
students in need of remediation by creating mobile apps for three of its most popular developmental 
education courses: 


e English 099 (Fundamentals of Composition), 
e Math 098 (Beginning Algebra I), and 
e Math 099 (Beginning Algebra II). 


This initiative builds upon BPCC’s prior development and implementation of Open Campus,’ which 
provides free and fully online access to developmental education course content to students anywhere 
in the world. 


BPCC’s current initiative to develop mobile app versions of this courseware stems from the twofold 
recognition that providing students with anywhere, anytime access to engaging curricular content can 
improve performance in courses, and that the student populations most likely to be assigned to 
developmental education have far higher rates of smartphone ownership than ownership of laptop and 
desktop computers. 


Additionally, BPCC proposed to build an integrated data platform (IDP), incorporating Open Campus 
and other student academic data, into a predictive flow model to reveal empirical patterns of student 
persistence and learning and to develop interventions targeted at students with low predicted 
probabilities of success. 


The US DOE awarded BPCC nearly $2 million through the FITW grant program due to the 
innovativeness of BPCC’s proposal as well as the strength of its proposed evaluation design. 


Impact Evaluation 


To determine the impact of the newly developed mobile app courseware on student outcomes, BPCC 
agreed to randomly assign developmental education course sections to either a /reatment group ot a control 
group. Specifically, course sections would be grouped or “blocked” based on the semester they were 
offered, the course number (Engl 99, Math 98, or Math 99), and instructor, and then course sections 
would be randomly assigned to the treatment or control condition within each block. This research 
design is a type of cluster randomized controlled trials called a multisite or blocked randomized design 
(Bloom, Bos, & Lee, 1999; Hedges & Hedberg, 2007; Murray, 1998; Murray, Varnell, & Blitstein, 2004). 


Students assigned to the treatment group would receive access to the mobile apps, whereas students in 
the control group would receive the “business-as-usual” condition of traditional in-person 
developmental education courses. 


By comparing the course performance and rates of persistence and attainment for the two groups, the 
impact evaluation was designed to provide evidence of the impact of the reform on students’ 


3 For more on BPCC’s Open Campus™, see http://www.bpcc.edu/opencampus. 
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postsecondary outcomes that meets the What Works Clearinghouse’s (WWC)* evidence standards 
without reservations, the highest WWC standard. 


Implementation Evaluation 


The purpose of the implementation evaluation was to assess the degree to which BPCC implemented 
the intervention with fidelity—in other words, whether they implemented the activities outlined in their 
FITW grant proposal—as well as to examine the contextual factors that facilitated or impeded 
implementation. 


Various methods were used to examine implementation. First, site visits were conducted to BPCC 
annually, including during both semesters of the intervention (Spring 2017 and Fall 2017). These site 
visits included semi-structured interviews with administrators, faculty, advisors, other student-facing 
staff, and students themselves. All interviews wete recorded and transcribed, and the text of the 
interviews was analyzed to identify key themes that emerged during the interviews. 


Second, the evaluator and the FITW team at BPCC participated in conference calls every 1-3 months to 
discuss progtess on various grant activities, such as the development of the mobile apps, the 
implementation of the IDP, training of BPCC staff to utilize both resources, and the like. Detailed 
notes were taken during these conversations, which formed the basis for much of the implementation 
evaluation. 


Third, various documents were reviewed that provided more context to the implementation of grant 
activities. For example, examples of materials that were used in professional development sessions with 
faculty and staff, one-page handouts for students that explained how to use the mobile apps, and 
similar documents were provided to the evaluation team by BPCC. 


Fourth, data on students’ use of the mobile apps was also analyzed to determine students’ depth of 
engagement with the intervention. Although these analyses did not form part of the impact evaluation, 
they provided additional context into the extent to which students’ were using the resource and factors 
that related to student usage. 


Impact Evaluation Sample 


The mobile apps were first made available to students during the Spring 2017 semester in Year 2 of 
grant implementation (2016-2017). As described in BPCC’s original FITW proposal, course sections 
for the three targeted developmental education courses were randomly assigned to the treatment or 
control group of students. The resulting sample for Cohort 1 included 867 unique students representing 
985 enrollments in one of the three courses (some students were enrolled in more than one of the 
targeted courses). In February 2017, the third-party evaluator conducted a site visit to interview 
students, faculty, and staff to understand how implementation was proceeding and to gauge 
stakeholders’ perceptions of the mobile apps. 


During Year 3 of the grant period, BPCC piloted the mobile apps with a second cohort of students 
who were enrolled in one of the three targeted developmental education courses during the Fall 2017 
semester. A blocked randomized design was once again used to estimate the effect of the mobile apps 
on student outcomes. The resulting sample for Cohort 2 included 1,051 unique students in 1,142 total 


* The Institute of Education Sciences’ What Works Clearinghouse is available at https://ies.ed.gov/ncee/wwe. 
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course enrollments. In October 2017, the third-party evaluator conducted another site visit to interview 
faculty, staff, and students to understand how implementation was proceeding. 


Across both interventions, the sample includes 2,036 unique students comprising 2,127 course 
enrollments. Analyses of outcomes specific to the developmental education courses, such as the grades 
students received in the courses and whether they passed the courses, used the full sample of course 
enrollments, allowing individual students to contribute multiple records to the estimation. Analyses of 
longer term outcomes, such as persistence to the next semester of college and credential attainment, 
used unique students and restricted the sample to only students enrolled in a single targeted 
developmental education course (Engl 99, Math 98, or Math 99) to ensure that students were not 
included in both the treatment and control groups. 


The remainder of this report is structured as follows. 


e The first section provides a more detailed description of the United States Department of 
Education’s First in the World grant program (US DOE’s FITW), including the goals of the 
initiative and expected activities and results for grantees. 

e The second section more fully describes BPCC’s FITW proposal. 

e The third section discusses the methods that were used for the implementation and impact 
evaluations. 

e The fourth and fifth sections present the results of the implementation and impact evaluations 
respectively. 


> 


e The sixth and final section concludes by highlighting key findings and identifying factors that 
appeared to be most critical to the implementation of BPCC’s FITW strategies. 
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US DEPARTMENT OF EDUCATION’S 
FIRST IN THE WORLD PROGRAM 


The FITW grant program is sponsored by the US DOE’s Fund for the Improvement of Postsecondary 
Education (FIPSE). The purpose of the FITW program is to: 


...support the development, replication, and dissemination of innovative solutions and 
evidence for what works in addressing persistent and widespread challenges in postsecondary 
education for students who are at risk for not persisting in and completing postsecondary 
programs, including, but not limited to, adult learners, working students, part-time students, 
students from low-income backgrounds, students of color, students with disabilities, and first- 
generation students. (FITW Program Description)’ 


As evidenced by the program’s name and its description, FITW is designed to invest in innovations in 
postsecondary education that have promise to significantly impact the likelihood of success for our 
most vulnerable student populations. 


Given the emphasis on innovations, FITW also requires that grantees contract with third-party 
evaluators to conduct a rigorous evaluation designed to provide strong evidence of the impact of the 
initiative. For that reason, FITW grantees generally commit to conducting experimental, and in select 
cases quasi-experimental, evaluation designs that have a strong likelihood of producing such evidence. 


FITW began awarding grants in the 2014 fiscal year and distributed approximately $75 million to 25 
applicants, which consisted of individual postsecondary institutions or consortia of colleges and 
universities. FITW awarded approximately $62 million to 18 grantees during FY 2015, for a total of 43 
grants that have been awarded to date. Given the high number of applicants and the relatively small 
number of grants awarded, FITW is a highly competitive grant program. 


> For more on the FITW program, see https://www2.ed.gov/programs/fitw/index.html. 
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BPCC’S CONTEXT AND FITW PROPOSAL 


BPCC, located in the Shreveport metropolitan area in northwest Louisiana, serves a large and diverse 
population of approximately 10,000 students annually. BPCC is also one of the fastest-growing 
community colleges in the US, with fall enrollment increasing from 4,845 students in 2005 to 8,695 
students in 2014. 


Demographically, slightly more than 40 percent of the student body is white and another 40 percent is 
black, with the remaining 20 percent of the student body comprising 7 percent or less each of Hispanic, 
Asian, multiracial, and students of unknown/other demographics. More than half of undergraduates 
received Pell grants, an indicator of financial need. 


Slightly less than half (48 percent) of the population is younger than 22, with high school students 
taking college courses making up roughly 10 percent of the student body. A quarter of students are 30 
years or older, and the remaining quarter are aged between 22 and 29. 


Approximately 75 percent of students come from either Bossier Parish or Caddo Parish, the 
neighboring Parish in which Shreveport is located. 


However, the remainder of the population consists of students from 60 other Parishes across 
Louisiana; in addition, a small percentage (<3 percent) of students are out-of-state enrollees. 


In short, BPCC embodies the community college mission of maintaining a strong focus on providing 
access to college for students in its local community while simultaneously serving students from a 
diverse array of geographic, racial, and socioeconomic backgrounds. 


The reform that serves as the foundation of BPCC’s FITW initiative is Open Campus, BPCC’s free, fully 
online suite of courseware for non-credit courses. BPCC faculty and administrators began discussing 
the idea of Open Campus in 2012 while deliberating upon two overlapping issues related to 
developmental education. 


e First, BPCC students in developmental education, like developmental education students across 
the country, had relatively low rates of success. Few of these students were able to get high 
marks in the courses on their first attempt, even fewer transitioned into and successfully 
completed credit-bearing courses, and still fewer were able to persist and attain a degree or 
certificate. 

e Second, faculty and administrators believed that many students were improperly assigned to 
developmental education courses in the first place and were dismayed at the percentage of 
students requiring remediation, as high as 57 percent of incoming students in 2014. Many 
students assigned to developmental education courses were adult learners who had been out of 
school for some time. Although they may have previously learned the content and developed 
the skills needed to succeed in credit-bearing courses, they were unable to score at the required 
levels on the placement exams without a refresher on the content covered in the exams. 


Open Campus was therefore devised primarily as a means of allowing students to brush up on core 
academic content in math, reading, and writing prior to taking the developmental education placement 
exam. The Open Campus courses were designed and built by faculty who traditionally taught in-person 
developmental education courses at BPCC. 


The Open Campus coursewate consists of video lectures, supplemental reading materials, and quizzes 
that provide immediate feedback on progress toward mastery of the content. Additionally, the content 
is modularized, or broken down into small and digestible topics, so students can access the content 
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either sequentially or topically, focusing on the knowledge and skills for which they need the greatest 
remediation. The primary goal was to ensure that students placed into the courses that were most 
appropriate to their skills and academic needs and that students were better prepared for success in 
those courses. 


However, as awareness of Open Campus began to grow at BPCC, faculty and administrators learned that 
a number of students were accessing Open Campus modules concurrently during their enrollment in 
traditional in-person developmental education courses. 


Many students reported the benefits of being able to rewatch lectures on particular topics that they did 
not sufficiently understand during class and retake quizzes to ensure that they grasped the material. 


Some faculty therefore began to recommend that students watch the online video lectures through Open 
Campus ptiot to class so they could more fully discuss questions students had about the materials and 
then work collectively on assignments in-person, in class. 


This pedagogical approach, in which students watch lectures at home and complete “homework” in- 
class, has been described as the “flipped” classroom or “blended learning” and is becoming an 
increasingly popular pedagogical approach, as evidenced by the burgeoning literature on the topic 
(Bersin, 2004; Garrison & Kanuka, 2004; Graham, Woodfield, & Harrison, 2013; Thorne, 2003). 


BPCC began piloting Open Campus in 2013 and has seen significant access to and engagement with the 
online curriculum in the past four years. Indeed, BPCC administrators have learned that students from 
other states, and even other countries, have been accessing Open Campus to brush up on their academic 
skills. Preliminary research on Open Campus has also shown positive results. 


A one-year (academic year 2014—2015) pilot program of Open Campus in desktop and laptop formats 
demonstrated measurable increases in the number of students who persisted in and completed 
developmental courses. Students reported that they valued extended opportunities to view lectures they 
may have missed, review difficult concepts, supplement their lecture notes, and prepare for exams. 


Students who are ill, active military or on-call/shift-workers benefit from the knowledge that they may 
recover any of the covered developmental courses’ lecture material missed due to excused absences. 
High school students preparing for placement tests and nontraditional students considering a return to 
college reported that they valued an opportunity to practice taking online tests and having portable 
topic-based tutorials in hand when needed. 


Additionally, the reach of Open Campus expanded, stemming from BPCC’s partnership with 
Northwestern State University of Louisiana (NSU). Located in neighboring Natchitoches, NSU is a 
four-year, public university founded in 1884 that currently serves approximately 9,500 students.° 


BPCC and NSU have established numerous transfer agreements, and one program, BPCC@NSU,’ 
offers students who are admitted to NSU—but who do not meet NSU’s college-readiness requirements 
in specific subjects—the opportunity to complete developmental coursework offered by BPCC onsite 
at NSU. 


Students who enroll in these developmental education courses at NSU also use Open Campus to 
supplement their in-person course experiences. However, due to recent changes in state policy that 


6 For more information on Northwestern State University of Louisiana, see https://www.nsula.edu/about-us. 


7 For more information on BPCC@NSU, see https://www.nsula.edu/ece/bpcc. 
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allow four-year colleges to offer developmental education courses, NSU may no longer be required to 
partner with BPCC in order to provide remedial coursework to students. 


Given the innovativeness and success of Open Campus, BPCC administrators involved with the initiative 
have been invited to present the Open Campus concept at a number of professional conferences: 


e the American Association of Community Colleges (AACC) Annual Convention (2013, 2014); 

e the American Association of Colleges and Universities (AAC&U) General Education and 
Assessment Conference (2013, 2014); 

e the National Academic Advising Association (NACADA) National Conference (2013); 

e the NACADA International Conference in the Netherlands (2013); 

e the League for Innovations in the Community Colleges International Conference (2014); 

e =the Sloan International Conference on Online Learning (2013); and 

e the National Institute for Staff and Organizational Development (NISOD) International 
Conference on Teaching and Leadership Excellence (2014). 


Although Open Campus has flourished by many measures over the past four years, one concern that 
remained about the availability of Open Campus was the fact that many of BPCC’s students, a significant 
proportion of whom can be considered low-income or the “working poor,” did not have access to 
computers or to reliable internet access at home. Students would frequently access Open Campus 
through the computers located in the campus library, but for many working adults and individuals with 
family responsibilities, this was not significantly more convenient than coming to campus for in-person 
courses. 


Understanding this issue, BPCC proposed in its FITW application to provide selected Open Campus 
developmental education courses on mobile application platforms for both iPhone and Android 
devices, capable of serving up to 10,000 students at a time on each platform. 


Three developmental education courses available through Open Campus were initially identified for 
conversion from desktop to mobile versions: Math 98 (Beginning Algebra I), Math 99 (Beginning 
Algebra ID), and English 99 (Developmental Writing/Fundamentals of Composition). 


These three courses were targeted because they have some of the highest enrollment among 
developmental education courses at BPCC. The mobile apps would be made available to students 
enrolled in the targeted developmental education courses at either BPCC or NSU (although 
BPCC@NSU coutses may no longer be offered given state policy changes allowing four-year college to 
provide developmental education instruction without the need to partner with a community college). 


After this initial rollout of three courses, additional courses may be selected for the creation of mobile 
apps. Whether this expansion occurs depends largely upon the results of the third-party evaluation 
aimed at understanding the degree of success of the initial courses and identifying the factors that were 
most critical to successful implementation. 


By end of the grant performance period, it is expected that the mobile apps will be scaled to a broader 
campus-wide design that would benefit all students, including underrepresented, underprepared, and/or 
low-income students. The mobile apps are also expected to be useful as a professional development 
tool to demonstrate effective pedagogical practices to new and adjunct BPCC faculty. 


In addition to developing the mobile apps, BPCC also proposed to implement an integrated data 
platform (IDP) that would incorporate multiple streams of student data into a predictive flow model to 
illuminate patterns of student persistence and learning. The institution-specific insights would be used 
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to inform interventions, policy decisions, and changes in practice that support student persistence and 
graduation. 


The build-out of the platform and the integration of other applications in use at BPCC, including 
Banner,* Canvas,’ institutional survey data, and tools developed by Civitas Learning,’ would help 
BPCC measure the effectiveness of the project interventions on various student subgroups and student 
outcomes. 


Long-term, the integration of data from developmental education students’ course performance and 
engagement with the mobile apps into the integrated data platform would enable better targeting of 

interventions at students with the greatest risk of dropping out of BPCC due to their performance in 
developmental education courses—thereby improving student persistence and graduation rates. 


8 Banner is enterprise resource planning (ERP) software from Ellucian; for more information, see 
https://www.ellucian.com/student-information-system. 


° Canvas is learning management system (LMS) from Intructure; for more information, see 
https://www.canvaslms.com/about-us. 


10 Civitas Learning, Inc., provides resources to higher education that help bring together diverse sources of data; for 
more information, see https://www.civitaslearning.com/about. 
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INDEPENDENT EVALUATOR 


This evaluation is being conducted by Giani Consulting & Evaluation, LLC (GCE). Matt Giani, PhD, is 
the firm’s principal and serves as the lead evaluator for BPCC’s FITW initiative. 


Dr. Giani received his PhD in education policy and planning from the University of Texas at Austin 
and holds a master’s degree from Stanford University in education policy, organization, and leadership 
studies. 


He previously served as an evaluator for three US Department of Labor Trade Adjustment Assistance 
Community College and Career Training (TAACCCT)" grants and was patt of the national research 
team for the foundation-funded Credit When It’s Due initiative’* studying the implementation and impact 
of reverse credit transfer policies in 15 states. 


Dr. Giani now serves as a research scientist for the Office of Strategy and Policy at the University of 
Texas at Austin.'® His research has appeared in outlets such as Review of Higher Education, Research in 
Higher Education, The Journal of Vocational Education and Training, and The High School Journal. 


Giani Consulting & Evaluation is the independent evaluator of BPCC’s FIT'W initiative. In this role, 
the firm is independently conducting all key aspects of the implementation and impact evaluation. 
Specifically, GCE is responsible for executing the random assignment of students to the control or 
treatment groups, collecting and analyzing student outcome data, and estimating and reporting program 
impacts on the student outcomes. 


GCE will also conduct an implementation evaluation to assess the extent to which BPCC implemented 
the activities it proposed in its FITW grant applications and to understand the factors that inhibited or 
facilitated implementation. Though the BPCC project director will have an opportunity to review and 
comment upon the evaluation findings, the findings will not be subject to the approval of the project 
grantee before being reported. 


GCE teceived Institutional Review Board approval from both BPCC and NSU (Northwestern State 
University of Louisiana). GCE has also established a Memorandum of Understanding (MOU) with 
BPCC for the transfer of student data for the purposes of the evaluation in compliance with all state 
and federal laws governing the privacy and confidentiality of student data, including the Family 
Educational Rights and Privacy Act (FERPA). 


1! Ror more on the US Department of Labor's Trade Adjustment Assistance Community College and Career Training 
(TAACCCT) program, see https://doleta.gov/taaccct. 


'2 For more on the Credit When It’s Due (CWID) project, see https://occrl illinois.edu/cwid/products and 
http://www.washineton.edu/cctri. 


13 https://education.utexas.edu/ departments /educational-leadership-policy/graduate-programs/education-policy- 
planning-program/alumni-profiles/matt-s-giani 
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METHODS 


As required by the US DOE and the FITW grant program conditions, the third-party evaluation of 
BPCC’s FITW initiative consists of both an implementation evaluation and an impact evaluation. The 
following sections describe the methods that were used in these two components of the evaluation. 


Implementation Evaluation 


Research Questions 

The implementation evaluation is guided by two broad research questions: 
1) To what extent did BPCC implement the various reforms that constitute its FITW initiative? 
2) What factors most influenced implementation? 


The first question draws heavily from the logic model BPCC developed to conceptualize the activities 
and intended outcomes of the initiative, shown in Table 1. This portion of the implementation 
evaluation is principally concerned with implementation fidelity—namely, to what extent BPCC 
undertook the activities it originally proposed in its grant application. 


Research questions related to this portion of the implementation evaluation take the form of inquiring 
as to whether specific activities listed in the scope of work have occurred (e.g., “Did BPCC identify and 
hire project personnel?”’). 


The second question is not limited to assessing the implementation of specific activities listed in 
BPCC’s FITW grant application, but rather is concerned broadly with why implementation proceeded 
as it did and what factors most influenced implementation. From this perspective, successful 
implementation depends upon the alignment and coordination of multiple actors and stakeholder 
groups at different levels of the campus—and both inside and outside BPCC. 


A useful heuristic is to consider the role and functioning of six separate factors or stakeholder groups: 
students, faculty, support and advising staff, FITW staff, BPCC administrators, and the broader social, 
economic, and policy environment. That is, for BPCC’s FITW initiative to succeed: 


1) Students must agree to use the project apps that are made available to them; 

2) Faculty must effectively incorporate use of the apps into their classroom instruction and utilize 
the IDP; 

3) Support staff must ensure that the apps are developed and working effectively and advising 
staff must also utilize the IDP; 

4) FITW staff must monitor the implementation of the apps, address concerns as they arise, and 
modify the tools to increase their efficacy; 

5) BPCC administrators must provide the structures, incentives, and opportunities necessary to 
support the implementation of the apps and IDP; and 

6) The broader social, economic, and policy environments must remain favorable for the initiative 
to be implemented, sustained, and scaled. 


This portion of the implementation evaluation therefore seeks to uncover and explore the 
contributions of these constituent elements of the BPCC ecosystem to the implementation of the 
FITW initiative. 
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Table 1: BPCC’s Logic Model for Its FITW Initiative 


Note: This BPCC logic model table has been lightly edited for clarity to readers outside of the BPCC community. 


ACTIVITY OUTCOME GOAL TIMELINE 
ldsatiey and Wave esjece pamonnel co to administer grant is built. Project fully staffed within 90 days of start Y1Q1 
Designate campus office space and 
purchase supplies for project Physical space is provided and offices are established for new hires within 90 days. Y1Q1 
personnel. 
Capacity to collect and track participant outcomes is increased. Servers are ordered 
Order and install equipment and Laas aes P ae ; 
sunolics and installed, workstation is installed, Mediasites hardware and media storage are Y1Q2-Y1Q3 
a fully operational by end of third quarter. 
; ee ; : — Three Open 

Procure consultant for app vendor Professional expertise is identified to inform the project. Vendor selection criteria 

Campus Y1Q2-Y1Q3 
selection process. are formulated. 

modules 
Contract with integrated data embedded 
int f 
pou GD) teeter cae Professional expertise is identified to inform the project. adele FG Q2-Y1Q3 
gathering features needed for app 
development. 
Procure subcontracted vendor for Professional expertise is identified to inform the project. Mobile app developer is 
Y1Q2-Y1Q3 
app development. procured; subcontract is developed by end of third quarter. 
Expand existing instructional modules 
for pilot population to mobile app Existing Open Campus templates are converted to formats compatible with mobile ¥1Q2-Y1Q3 
format for all FTIC (first time in devices. 7 
college) population. 
: Mobile-device app requirements are identified; conceptual art, wireframes, and New mobile 

Develop mobipapp coaccprane designs developed and finished; domain name and server space allocated; working | app scaled to | Y1Q3-Y1Q4 


begin work with developer. 


timeline established. 


dev ed, 
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ACTIVITY OUTCOME GOAL TIMELINE 
ae ap ales ‘ 4 student 
Convert existing modules to mobile Availability of tutorial support for students is scaled. Existing modules are siccessand: || VIO W204 
app format. converted and available for pilot use by end of first year. : 
gateway 
Incorporate instructional materials Existing instructional materials are incorporated into mobile app format by end of | courses Y1Q4 
into mobile app. first year. 
Identify target population for pre- Experimental and control groups for beta testing are identified from a universe of Y2Q1 
project placement testing. all students enrolled that semester. 
Ccdece Baseline ce RAeaiGe: Capacity to collect and compare valid data is established. Initial untreated Y2Q1 
measurements ate gathered among target population before beta testing begins. 
Solicit student feedback on design, . , 
: - Formative evaluation begins. Through focus groups, surveys, and interviews, 
levels of confidence/anxiety, usability | . sales Y2Q2 
: information is collected about student perceptions and use of app. 
of mobile app. 
Share beta version of mobile app with | Formative evaluation feedback continues as the beta version goes live; Initial tests 
: ae : Y2Q3 
selected group of students, public. will inform future refinements of the mobile app. 
Collect data on mobile app design - : seed : 
fisabilien, ehallbapes audistrenothoor Initial data collected; by end of second year, modifications made to mobile app ¥2Q3-Y204 
beta version to reflect predominant user preferences. 
features. 
ors eo stnuosdce caer age Mobile app features are refined based on feedback and beta test results through 
improvement loop to refine mobile Paedecae Y2Q4-Y3Q4 
app. oe Continuously 
; ; P refined 
wer alae (0 salcessic nopue Increase number of faculty and staff who attend professional development events | mobile app 
app in standard developmental ed : Y2Q1-Y4Q1 
related to Open Campus mobile app implementation. that is 
math course curricula. ; 
available for 
Transition funding of new hire for all students. 
app administration to college Continued progress following grant period; the work is sustained. Y4Q4 


operational funds. 
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Research questions that fall within this portion of the implementation evaluation include: 


1) To what extent did social, economic, and policy changes influence implementation? 
2) How supportive was BPCC administration of the FITW initiative? 
3) Who was part of the FITW support team, and what roles did they play in supporting the 
initiative? 
4) How did support and advising staff at BPCC contribute to the development and implementation 
of the mobile apps and integrated data platform (IDP)? 
5) How did faculty respond to and embrace the FITW initiative, and what training and professional 
development opportunities were provided to them in order to increase their support? 
6) To what extent did students use the mobile apps, and what factors influenced their degree of 
utilization? 
7) Overall, what were the greatest barriers to and facilitators of implementation at BPCC? 
In the first year of the FITW grant (2015-2016), BPCC stakeholders focused their efforts on developing 
the mobile apps and communicating the design and purpose of the initiative to various constituencies on 
campus but did not begin implementing the intervention this year. The implementation evaluation in the 
first APR therefore focused on BPCC’s social, economic, and policy contexts and various stakeholders’ 
perceptions of the initiative. The remaining APRs have focused largely on the implementation of 
activities proposed in BPCC’s FITW grant application and factors promoting or inhibiting successful 
implementation. 


Implementation Evaluation Methods 


Four primary methods were used to evaluate the implementation of the mobile apps at BPCC. First, the 
third-party evaluator conducted annual site visits each year of the grant. During site visits, interviews and 
focus groups were conducted with BPCC administrators, FITW staff, advising staff, instructors who 
were teaching one of the three developmental education courses included in the intervention, and 
students enrolled in one of the targeted courses. Prior to the site visit, separate semi-structured focus 
group protocols were developed for each of the stakeholder groups. All focus groups were recorded and 
professionally transcribed. Responses for each focus group were then coded and themed, and responses 
between focus groups were triangulated. 


Second, the evaluator and BPCC developed surveys to gauge input from broader groups of stakeholders 
on various topics related to implementation. For example, BPCC developed and administered feedback 
surveys for faculty and staff who participated in professional development sessions related to both the 
mobile apps and the IDP, and the independent evaluator administered student and instructor surveys to 
gauge their perceptions of the mobile apps. However, response rates to the survey were generally quite 
low for both the student and faculty/staff surveys, preventing the survey data from producing reliable 
insights into implementation. 


Third, the evaluator held regular phone calls with FITW staff at BPCC throughout the year to gauge how 
implementation was proceeding. These calls occurred on a roughly monthly basis. FITW staff also 
provided the evaluator with various documents related to implementation, such as training and 
professional developmental materials, handouts for students that explained how to use the mobile apps, 
and related materials. 


Fourth, the evaluator reviewed data on students’ usage of the mobile apps. While these analyses did not 
form part of the impact evaluation, they provided additional evidence as to the depth of students’ 
engagement with the mobile apps and factors related to students’ usage of the apps. 
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Impact Evaluation 


Impact Evaluation Research Questions 


The overarching research question guiding the impact evaluation can be stated as: What was the impact 
of access to the mobile applications on the postsecondary outcomes of developmental education 
students? The outcomes analyzed in the impact evaluation include short- (e.g. course performance), 
medium- (e.g. semester-to-semester persistence), and long-term (e.g. postsecondary attainment) 
outcomes. BPCC’s FITW grant was awarded in 2015 and the four-year grant period ended on September 
30, 2019. Student outcomes were therefore tracked through Spring 2019, or approximately two full years 
from the first semester of implementation of the mobile apps. For all primary research questions, a sub- 
question investigates the extent to which the effect of the mobile apps varied across the three courses— 
in other words, whether courses moderated the effect of the app on student outcomes. 


Final Annual Performance Report—Combined Cohorts 


1) What is the effect of access to mobile apps on the rates of students passing one of the three 

targeted developmental education courses (English 99, Math 98, and Math 99)? 
a. Does this effect vary by course number? 

2) What is the effect of access to mobile apps on the grades students receive in the three targeted 

developmental education courses (English 99, Math 98, and Math 99)? 
b. Does this effect vary by course number? 

3) What is the effect of access to mobile apps on the semester-to-semestet persistence of students 
enrolled in one of the three targeted developmental education courses (English 99, Math 98, and 
Math 99)? 

c. Does this effect vary by course number? 

4) What is the effect of access to mobile apps on the next semester GPA of students enrolled in one 

of the three targeted developmental education courses (English 99, Math 98, and Math 99)? 
d. Does this effect vary by course number? 

5) What is the effect of access to mobile apps on the overall course passing rates during the next 
semester of students enrolled in one of the three targeted developmental education courses 
(English 99, Math 98, and Math 99)? 

e. Does this effect vary by course number? 


6) What is the effect of access to mobile apps on the credits earned during the next semester of 
students enrolled in one of the three targeted developmental education courses (English 99, Math 
98, and Math 99)? 

f. Does this effect vary by course number? 

7) What is the effect of access to mobile apps on the credential attainment rates of students who 
enrolled in one of the three targeted developmental education courses (English 99, Math 98, and 
Math 99)? 

a. Does this effect vary by course number? 


Impact Evaluation Methods 


To produce evidence of the effects of this intervention that meets the What Works Clearinghouse’s 
(WWC) Evidence Standards without reservations, students were randomly assigned to either the 
treatment or the control group using a stratified cluster randomized trial research design, also known as a 
multisite or blocked randomized design (Hedges & Hedberg, 2007; Murray, 1998; Murray, Varnell, & 
Blitstein, 2004). We elected to use a group random assignment rather than individual assignment given 
the threat of “contamination,” namely students who would be given access to the mobile apps sharing 
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access with control students enrolled in the same developmental course sections. Additionally, faculty 
members who will be teaching the developmental education courses expressed a strong preference for 
cluster randomization to eliminate the need to assist students in the same classrooms using different 
materials and technologies. 


The four strata used in the randomization are cohort/semester (Spring 2017 or Fall 2017), college (BPCC 
ot NSU), developmental education course numbers (Math 98, Math 99, and English 99), and instructors. 
For every instructor who taught more than one course section of the same course number at the same 
college in the same semester, that instructor’s specific course sections were randomly assigned to the two 
groups. The remainder of the course sections, or those taught by instructors who only taught one course 
section of the same course number at the same college in the same semester, were then randomized 
within course number within college. The result was 17 instructor by course number by college blocks 
during the Spring 2017 intervention and 14 blocks during the Fall 2017 intervention for a total of 31 
blocks, each of which contained a treatment and control group. These blocks, and the number of 
students assigned to the treatment and control groups within each block, are reflected in Table 2. 


Table 2: Student Enrollment by Randomization Block for Both Cohorts 


Block ID Cohort BPCC/NSU Course Instructor Control Treatment ‘Total 


1 Spring 2017 +=BPCC ~~ Engl99 Engl99A 18 23 41 
2 Spring 2017 =BPCC ~~ Engl99 Engl99B 30 29 59 
3 Spring 2017  =BPCC ~~ Engl99 Engl99C 18 27 45 
4 Spring 2017 =BPCC ~~ Engl99 Engl99D 22 24 46 
5 Spring 2017  =BPCC — Engl99 Engl99E 28 26 54 
6 Spring 2017 NSU Engl99 Engl99F 14 19 33 
7 Spring 2017. =BPCC — Math98 Math98A — 29 24 53 
8 Spring 2017. =BPCC  Math98 Math98B 37 32 69 
9 Spring 2017. =BPCC — Math98 Math98C 29 27 56 
10 Spring 2017. =BPCC ~ Math98 Math98D 24 29 53 
11 Spring 2017. =BPCC = Math98 Math98E 23 32 55 
12 Spring 2017. =BPCC — Math98 Math98F 56 30 86 
13 Spring 2017 NSU Math 98 Math98G 13 16 29 
14 Spring 2017. =BPCC —  Math99 Math99A 34 28 62 
15 Spring 2017. =BPCC ~~ Math99 Math99B 33 33 66 
16 Spring 2017. =BPCC  Math99 Math99C 87 56 143 
17 Spring 2017 NSU Math99 Math99D 22 8 30 
18 Fall 2017 BPCC — Engl99--Engl99G 31 31 62 
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19 Fall 2017 BPCC Engl 99 Engl 99H 31 31 62 
20 Fall 2017 BPCC Engl 99 = Engl 99 I 31 a2 63 
ZL Fall 2017 BPCC Engl 99 = Engl 99 J 31 20 51 
22, Fall 2017 BPCC Engl 99 Engl 99 K 30 62 92 
23 Fall 2017 BPCC Math 98 Math 98H a0 3D 70 
24 Fall 2017 BPCC Math98 Math 981 35 35 70 
25 Fall 2017 BPCC Math 98 = Math 98 J 31 30 61 
26 Fall 2017 BPCC Math 98 Math 98K 34 11 45 
ZY Fall 2017 BPCC Math 98 Math 98 L 35 22 57 
28 Fall 2017 BPCC Math 98 Math 98 M 57 67 124 
29 Fall 2017 BPCC Math99 Math 99 E G2 35 107 
30 Fall 2017 BPCC Math 99 Math 99 F 35 35 70 
31 Fall 2017 BPCC Math99 Math 99G iS, 129 208 
Total 1084 1038 2192 


A master list of all course sections for the three targeted developmental education courses was generated 
near the beginning of the Spring 2017 and Fall 2017 semesters. These course sections were provided to 
the third-party evaluator, who conducted the randomization. This randomization occurred on Saturday, 
January 21, 2017, after the last day that students were able to register for new courses or change course 
sections (Friday, January 20, 2017) for the Spring 2017 intervention. The timing of the randomization 
eliminated the threat of joiners (students being added to the sample after randomization) and mitigated 
the risk of contamination (students moving from one assignment to another by changing course 
sections). 


However, this date was roughly a week after the Spring 2017 semester had begun, making it a challenge 
for instructors to effectively implement the mobile apps. For the Fall 2017 cohort, the randomization 
occurred on Sunday, August 6, 2017. This allowed instructors to be notified of whether they would have 
access to the apps in their classes during the professional development sessions that occurred on August 
7-8, 2017. The first day of class was on August 11, 2017. Although students may have enrolled in Fall 
2017 courses through August 17, 2017, no joiners were included in the analysis. 


Statistical Models 


As the use of a cluster randomized design has the potential to result in biased estimates of the standard 
error of the treatment effect stemming from the introduction of level-2 clustering (Hedges & Hedberg, 
2007), multilevel modeling techniques were used to account for this clustering (Raudenbush & Bryk, 
2002) with students nested in course sections. Multilevel linear regression models were used for both 
continuous and dichotomous outcomes. For dichotomous outcomes, linear regression was used rather 
than logistic regression both to facilitate interpretation and because the outcomes under study have 
moderate probability ranges (20-80%) that make linear regression an appropriate choice (Hellevik, 2009). 
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Two statistical models were used. The first included fixed block effects and a fixed treatment effect to 
estimate the average treatment effect across blocks. The statistical equation for the model may be 
described as: 


Model 1 


Level-1: Student Level 


M 
Yij = Boj + > BimXmij + €ij 
m=1 


Level-2: Cluster (Course Section) Level 


P-1 
Boj = You F Yo1(T;) + » Yo2pBlocky; + Uoj 
p=1 
Bimj = Y1.m0 
Where, 
Yij = the outcome for the 7 student in the 7" course section. 
Bo j = the intercept for course section /. 
Bim j 7— the effects of student covariates in course section /. 
Xmij = the m” of M additional covariates for student / in course section /. 
Ej = a residual error term for student 7 in course section /. 
Yoo = the mean intercept 
Yo1 = the treatment effect 
T; = 1 if course section / is assigned to treatment, and = 0 if course section / is assigned to 
comparison. 
Yimo = mean effect of student covariate m. 


Blocky; = 1 if the course / was assigned to the treatment or comparison condition within the 
randomization block p, and = 0 otherwise. 


Yo2.p = the effect of block p. 


Hoj = random intercept term — deviation of course section /s mean from the grand mean, 
conditional on covariates; assumed to be normally distributed with mean 0 and variance 
2 
Too: 


The second model estimated the extent to which the effect of treatment varies across course numbers by 
adding block by treatment interaction terms. The equation for this model can be described as: 


Model 2 


Level-1: Student Level 
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M 
Yij = Boj + >» BimXmij + €ij 
m=1 


Level-2: Cluster (Course Section) Level 


P-1 P-1 
Boj = Yoo + > Yo2pBlocky; + » Yosp(j * Blocky;) ar Ho; 
p=1 Dp 


Bij = Y10 


Bim = Y2.mo 


Where, 


Block; = 1 if the course section / was assigned to the treatment or comparison condition within 
the (randomization or matching) block p, and = 0 otherwise. 


Yo2zp = the effect of block p 


Yo3.p — the difference in the treatment effect for block p and the treatment effect for the 
reference block. 


All other terms are defined as described in the previous model. 
Samples 


This Annual Performance Report includes analyses of both cohorts of students (Spring 2017 and Fall 
2017). For the Spring 2017 cohort, the total number of students who were enrolled in one of the targeted 
courses at the time of randomization included 867 unique students in 985 total course sections. Of the 
867 unique students, 758 only enrolled in one of the three courses, 100 enrolled in two courses, and 9 
enrolled in all three courses. Five of these students did not receive a grade for their developmental 
education course for an attrition rate of 0.6%. Of the remaining 980 course enrollments, 463 (47.2%) 
were in treatment group course sections and 517 (52.8%) were in control group classrooms. Of the 757 
students who enrolled in only one course, 366 (48.3%) enrolled in a course section assigned to the 
treatment group and 391 (51.7%) enrolled in a course section assigned to the control group. 


For the Fall 2017 cohort, the total number of students who were enrolled in one of the targeted courses 
at the time of randomization included 1,051 unique students in 1,142 total course sections. Of the 1,051 
unique students, 960 only enrolled in one of the three courses and 91 enrolled in two courses. No 
student enrolled in all three courses. 


The randomization produced a relatively balanced distribution of assignments, with 575 course 
enrollments (50.4%) in the treatment group and 567 (49.6%) in the control group. Among the sample of 
960 students with only one course enrollment, 482 students (50.2%) were assigned to the treatment 
group and 478 (49.8%) were assigned to the control group. No joiners, or students who enrolled after the 
time of randomization, were included in the analysis. 


Although randomization occurred after the last date for course changes for the Spring 2017 intervention, 
randomization took place slightly earlier in the Fall 2017 semester in order to give instructors more time 
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to prepare for using the apps in their courses (or not, depending on their assignment). While only five of 
the 985 student course records did not have credit or grade information for their assigned developmental 
education course at the end of the semester during Spring 2017 (attrition rate = 0.6%), for the Fall 2017 
semester 112 of the 1,142 course records did not have a grade (attrition rate = 9.8%). The attrition rate 
for the control group was 10.9% and the rate for the treatment group was 8.7%, for a differential 
attrition of 2.2%. 


Neither the overall or differential attrition rate poses a threat to the validity of the study under WWC’s 
conservative attrition standard. Course records without a grade will be considered censored and excluded 
from the grade analyses per IES recommendations (Puma, Olsen, Bell, & Price, 2009). However, for the 
analysis of whether students passed the course, we will consider students who withdrew or otherwise did 
not receive a grade as having not passed the course, and these records will be kept in the analytic sample. 


The analytic sample used in each model depends upon the research question being addressed. For 
models of outcomes that are specific to the individual developmental education courses, the entire 
sample of was be used given the limited threat of contamination. For example, it is unlikely that a student 
getting access to the mobile app for Math 98 would substantively affect her performance in English 99, 
even if she was assigned to a control group section in English. 


However, for outcomes related to overall course performance and longer-term outcomes, an additional 
inclusion criterion is applied requiring students to have only enrolled in one of the three targeted course 
sections in that semester to ensure students are not part of both the treatment and control group." 


Variables 


The models include student-level covariates (race/ethnicity, gender, Pell receipt, age, credits attempted 
prior to the semester of the intervention, and credits earned prior to the semester of the intervention) as 
well as a level-2 random intercept to account for variation in the effects of classtooms/instructors on 
student outcomes. Although standardized assessments were administered to students, no more than 40% 
of the sample took the same assessment, and roughly 10% of the sample did not have a score for any 
standardized assessment. For this reason credits attempted and earned prior to the intervention semester 
were used as a proxy for academic preparedness. Fixed effects for the 31 college by course number by 
instructor blocks were added to the model to account for the stratified cluster randomized design. 


'4 Note that the eligibility criterion (that student enrolled in only one course) for inclusion in this sample is based on a 
student behavior that occurred prior to randomization, maintaining the integrity of the random assignment. 
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IMPLEMENTATION EVALUATION RESULTS 


As discussed in the methods section, the implementation evaluation has two parts. The first can be 
described as the implementation fidelity evaluation. This portion of the evaluation is principally 
concerned with the extent to which BPCC implemented the activities that comprised the reform’s logic 
model as listed in BPCC’s FITW proposal, or whether activities were implemented. The second examines 
the implementation context, including the current and potential roles of various BPCC stakeholders in 
implementing the initiative and critical factors inhibiting or facilitating implementation. This portion of 
the evaluation assesses ow and why BPCC implemented the strategies it did. 


Implementation Fidelity Evaluation 


The implementation fidelity evaluation focuses on those activities that were implemented by the end of 
the fourth grant year, or through September 2019. Additionally, the description of implementation 
activities focuses on those activities which were proposed to continue through at least the third grant 
year. For example, the activity “order and install equipment and supplies [and] hire consultant for 
application vendor selection process” was an activity that only occurred in Year 1. In contrast, the 
activity “continuous feedback improvement loop; refine mobile app” was proposed to begin in Year 1 
and continue throughout the grant. The implementation of this activity is therefore discussed in greater 
detail. BPCC in fact proposed no new activities to begin in Year 3 or Year 4, meaning the 
implementation evaluation focuses on the continuation of activities begun in Year 2 through the end of 
the grant. 


Table 3 contains a summary of the implementation fidelity evaluation. The table lists each activity in 
BPCC’s logic model the college indicated it would implement during Years 1-4 of the grant. Table 3 also 
includes two sets of ratings for each activity. The initials “MG” refer to the independent evaluator’s 
ratings of implementation, while “AM” refers to the project director’s ratings of implementation. The 
columns in the table represent six categories of implementation. Definitions of these categories are as 
follows: 


e Discontinued — An activity that was planned in the original FITW grant application is no longer 
being planned. 

e Planning — Project personnel are in the planning stages of an activity, with implementation to 
follow in the near future. 

e Implementing — Implementation has commenced for the given activity but has not yet been 
completed. 

e Implemented — The activity has been fully implemented by the college, and no additional actions 
related to that activity are planned or required. 

e Sustainability — A formal plan has been put in place for the activity to continue beyond the 
termination of the grant period. 

e Scale-up — The activity is being expanded beyond its original scope as described in the grant 
proposal. 


Only some activities were intended to be sustained or scaled after the end of the grant. For example, the 
contract with the vendor to develop the mobile apps was intended to occur during the grant period but 
not extend after the grant. In contrast, BPCC planned to transition staff hired on grant funds to more 
sustainable revenue sources in order to keep these personnel on staff. The discussion below will only 
focus on sustainability and scale-up for those activities that were intended to be sustained. 
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Table 3: Evaluation of Implementation of Years 1 and 2 Activities Included in BPCC’s Logic Model 
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Year 1 (2015-16) 
MG 
Identify and hire project personnel 
AM 
Designate campus office space and purchase MG 
supplies for project personnel AM 
MG 
Order and install equipment and supplies 
AM 
MG 
Procure consultant for app vendor selection process. aay 
Contract with IDP to determine data gathering MG 
features needed for app development AM 
MG 
Procure subcontracted vendor for app development AM 
Expand existing instructional modules for pilot MG 
population to mobile app format for all FTIC 
population AM 
Develop mobile app concept and begin work with MG 
developer AM 
MG 
Convert existing modules to mobile app format ve 
Incorporate instructional materials AM | MG 
Year 2 (2016-17) 
Identify target population for pre-project placement MG 
testing AM 
MG 
Conduct baseline data collection 
AM 
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Solicit student feedback on design, levels of MG 

confidence /anxiety, usability AM 

Share beta version with selected group of students, MG 

public AM 

Collect data on design usability, challenges, and MG 

strengths of features AM 

Continue continuous feedback and improvement MG 

loop to refine mobile app AM 

Work with faculty to integrate mobile app in MG 

standard dev ed math course curricula AM 

Transition new hire app admin to college MG 

operational funds AM 


Contract with Integrated Data Platform (LDP) 


As discussed in prior Annual Performance Reports, BPCC contracted with Civitas Learning to begin 
implementation of three components of the IDP: a base analytics platform called [ume that uses 
predictive modeling to identify at-risk students, a tool called Inspire (formerly known as Inspire for 
Advisers) that serves as a platform for targeting interventions and outreach to at-risk students, and a 
third tool called [Hume Impact that enables colleges to estimate the effect of specific programs and 
interventions on student outcomes by selecting a matched control group. BPCC proposed for the FITW 
grant funds to cover the costs of Inspire, and BPCC administration agreed to fund the Ilume and Illume 
Impact tools. 


A variety of technical challenges, including the centralization of data collection and management systems 
by the Louisiana Community and Technical College System (LCTCS) and the development of a 
dashboard view within Banner, the student data system used by LCTCS colleges, impeded 
implementation of the Illume and Illume Impact aspects of the IDP. However, BPCC fully implemented 
the Inspire for Advisers tool. By the close of Year 2, a total of 150 BPCC personnel — 76% of all BPCC 
full-time faculty and 52% of front-line staff — had been trained in IDP features most applicable to each 
departmental area. 


Training on the Inspire tool continued in Years 3 and 4 of the grant. By the end of the final grant year, 
more than 75% of faculty and 75% of front-line advising staff had been trained on the use of the tool. 
Approximately 100 faculty and staff used the tool independently after they received training, resulting in 
roughly 2,000 individual logins by faculty and staff and more than 11,000 outreaches to students. 


Continuous feedback improvement loop; refine mobile app 
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The first phase of the mobile apps was designed to make a number of content modifications to the 
resource and convert the desktop version of Open Campus into a mobile responsive and adaptive version 
of the tool. The content modifications focused on shortening the pre-recorded lectures from an average 
of 15—20 minutes to 2—3 minutes. 


This modularization was planned given early feedback from students and instructors that the videos were 
too long, as well as analytics from the videos themselves that showed viewers often did not watch the full 
videos before exiting. This modularization was completed in the first two years of the grant, and the 
Phase 1 version of the app that launched in grant Year 2 included these shortened videos. Student and 
instructor feedback and analytics from the videos suggest this strategy was effective, as viewers were 
much more likely to complete the shortened videos than the previous longer versions. 


However, the Phase 1 version was not a full mobile application that students could download from 
services such as the Apple App Store or Google Play and install on their devices. Rather, a short-cut to 
the mobile-friendly version of Open Campus could be installed on the home screen of students’ mobile 
phones. While installing Open Campus in this way made the tool resemble and function like a mobile app, 
both students and instructors were less familiar with this process than downloading apps through 
traditional channels. 


Additionally, Phase 1 of the mobile apps did not have functionality that would allow instructors to login 
and view their students’ engagement with the apps. Rather, BPCC administrators were required to send 
instructors weekly snapshots of student engagement, including which students logged into the tool and 
how they used it. Instructors reported during the Year 2 site visit that the lack of this functionality 
hindered their support for the tool. 


In Year 3 of the grant, BPCC contracted with InsiteHub, the same vendor that developed the first phase 
of the apps, to develop fully functional mobile apps that can be downloaded through the App Store and 
Google Play. These Phase 2 mobile apps were rolled out to BPCC students, faculty, and staff in the final 
year of the grant and are currently available through the Apple App Store and Google Play. The new 
features of the Phase 2 apps and how students and faculty responded to them will be discussed further in 
the implementation context evaluation section. 


Work with faculty to integrate mobile app in standard developmental education math courses 


One aspect of sustainability of the mobile apps consists of instructors fully integrating the mobile apps 
into their courses. This may consist of faculty assigning students videos to watch for homework, 
requiring them to complete quizzes within the mobile apps, among other activities. In order to maintain 
faculty buy-in for the FITW initiative and ensure instructors did not perceive the reform as a top-down 
mandate, BPCC administrators were flexible during the pilot periods in regards to how instructors could 
use the apps in their classrooms. However, they simultaneously recognize that the usefulness of the apps 
will be limited if instructors solely use them as supplemental, and optional, resources for students. 


After Phase 2 of the mobile apps was rolled out in the final year of the grant, BPCC staff and 
administrators provided additional professional development to faculty on how to effectively incorporate 
the apps into their courses. However, the integration of the mobile apps into the core teaching and 
learning experience of developmental education courses is still ongoing. The most common use of the 
mobile apps continues to be as a supplemental resource offered to students before taking their 
developmental education placement exams or mentioned at the beginning of the semester in 
developmental education courses. The following section provides additional detail regarding the factors 
that facilitated and impeded faculty and student use of the mobile apps in developmental education 
classrooms. 
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Implementation Context Evaluation 


The previous section was focused principally on fidelity of implementation, namely whether BPCC had 
implemented the various activities that comprised the initiative’s logic model included in its original 
FITW grant application. The current section dives deeper into the contextual factors of the intervention 
in order to explore how the initiative was actually implemented and why BPCC experienced the 
challenges and successes it did during implementation. This section draws more extensively on the 
findings from the focus groups and interviews conducted during the site visit, as well as regular phone 
conversations with BPCC staff overseeing the FITW initiative. The research questions that were focused 
on during the implementation context evaluation were: 


1) How did support and advising staff at BPCC contribute to the development and implementation 
of the mobile apps and IDP? 

2) How did faculty respond to and embrace the FITW initiative, and what training and professional 
development opportunities were provided to them in order to increase their support? 

3) ‘To what extent did students utilize the mobile apps, and what factors influenced their degree of 
utilization? 

4) Overall, what were the greatest barriers to and facilitators of implementation of the mobile apps 
and IDP at BPCC? 


Support and Advising Staff's Contributions to Development and Implementation 
of Mobile Apps and Integrated Data Platform 


Format of the Mobile Apps 


A key feature of the phase one version of the mobile apps is that they were not available for download 
through app repositories built into most mobile phones, such as the App Store (for Apple devices) or 
Google Play (for Android devices). Instead, BPCC developed a new website to host this version of Open 
Campus. The website was developed with a responsive design that scales automatically to the screen size 
of the device being used, a key difference from the desktop version of Open Campus. After opening the 
site on a mobile device and logging in, the user can save a shortcut to the site on their phone. 


Once that step is completed, the resource functions very similarly to a mobile app. However, instructors 
and students were somewhat unfamiliar with this process, and there were strong recommendations for 
BPCC staff to develop a fully functional mobile app. In Year 3 of the grant, BPCC staff contracted with 
a vendor (InsiteHub) to develop Phase 2 of the apps. In addition to adding various features to the 
software, this version of the apps will also be available to download through app repositories such as the 
App Store and Google Play. In interviews, instructors reported strong approval of BPCC staff's strategy 
to develop full mobile apps. 


Training and Professional Development 


A select group of faculty members were instrumental in developing the content for the Open Campus 
mobile apps. These instructors had been receiving updates on the progress of the apps and had been 
testing functionality in the apps throughout the fall semester. The majority of faculty members did not 
receive substantive training on the mobile apps until a professional development session immediately 
preceding the Spring 2017 semester. Although holding the training so close to the beginning of the 
semester was a challenge, faculty reported positive experiences of the session. One instructor accounted 
her recollection of that training: 


What they showed us in the professional development or the workshop, it was enough to give 
you an idea of how helpful it was going to be, and so I think we were all pretty excited about it. 
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It's just, like I said, now trying to incorporate everything or change things to make it fit was the 
only issue. 


BPCC staff overseeing the FITW initiative also incorporated training on the mobile apps into the 
professional development sessions held before the Fall 2017 semester. This training addressed concerns 
that faculty members had raised during the Spring 2017 intervention period, such as allowing instructors 
hands-on time using the apps, providing them with additional examples of how they could incorporate 
the apps into their instruction, and informing them earlier of which course sections would be assigned to 
the treatment and control groups so they could have additional time to prepare if their section was 
selected for implementation. This professional development session was well-received by the faculty. 


In regard to the IDP, more than 75% of faculty and advising staff had received training on Inspire by the 
end of the third year of the grant. Responses to post-training surveys indicated that the vast majority of 
participants felt that the trainings were useful. In addition, participants in the training reported that the 
faculty members who led the training were highly effective. As one faculty member stated: 


[The faculty member leading the training] was very passionate. He was sold on it and I had the 
utmost respect for him, which gave me good cause to say, this is something that I need to zero in 
on a lot more... He convinced me of how it was meeting some good needs that we need to 
consider as a school. 


Generating Student Buy-In 


After the course sections had been randomized to the treatment and control groups, BPCC staff sent 
students enrolled in treatment sections an email notifying them of their ability to download a mobile app 
for the course in which they were enrolled. Students’ email addresses served as their account name, and a 
default password had been set for them. The email explained that students would be prompted to reset 
their password during the first login. 


In addition to this email, BPCC staff created a one-page handout that explained in detail how students 
could log on to the site, reset their password, and install a shortcut to the site as an app on their device. 
Faculty distributed these handouts immediately after students were notified via email that the apps were 
available. Students interviewed in focus groups recalled receiving these handouts and described them as 
helpful and informative. 


After the initial communications with students, BPCC staff monitored individual students’ usage of the 
apps to determine the percentage of students who were using them and the extent of their engagement. 
BPCC staff sent follow-up emails throughout the semester reminding them of the resource and 
encouraging them to use it. The first email was sent roughly five days after the first email, and additional 
emails were sent every 2-4 weeks thereafter. The mobile apps were also designed to send students 
positive feedback via email when they completed certain milestones, such as completing certain 
percentages of quizzes or videos. 


Finally, BPCC staff overseeing FITW implementation scheduled in-person demonstrations of the apps in 
course sections assigned to the treatment group. The staff who conducted the demonstrations indicated 
that they were well received by both students and faculty. An instructor echoed this sentiment: 


I was very glad to have [a BPCC staff member visit the class]. [He] came over and helped 
students sign in, who I knew that there were a few who either had never signed in up to that 
point or had not signed in in a while. And so it was nice having him on hand. 


Generating Faculty Buy-In 
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Although BPCC staff overseeing FITW implementation communicated directly with students via email 
to inform them of the opportunity to download the apps, they knew that students were less likely to use 
the apps without active support and encouragement from their instructors. However, they also 
recognized that implementation posed a number of potential challenges for faculty, such as having to 
quickly learn the new mobile app technology and incorporate the apps into their instruction. 


In addition to the formal professional development session, BPCC staff identified a number of key 
faculty members in each department that were highly regarded by their colleagues to serve as the liaisons 
between project staff and the faculty. These faculty liaisons facilitated many of the professional 
development sessions and provided one-on-one coaching with other faculty using the apps. It appears 
that this strategy has proven effective at increasing buy-in for the initiative, as many instructors view the 
initiative as faculty-led rather than a top-down mandate from central administration. 


Finally, BPCC staff have discussed plans for creating short modules or webinars designed to explain to 
faculty how the apps work and how they can incorporate the apps into their lesson plans and course 
assignments. These modules would be co-created by faculty liaisons and BPCC staff directing the project. 
Additionally, these modules would explain to faculty the cognitive science principles underlying the apps, 
such as the benefits of information recall for cementing long-term learning and the “flipped classroom” 
approach to instruction. 


Advising Staff's Perceptions of IDP 


The majority of advising staff spoke highly of the usefulness of the Inspire tool. There were three key 
benefits mentioned by multiple advisors who participated in interviews. The first benefit was the ability 
to see all the interactions a student has had with different offices, divisions, and personnel at BPCC. This 
documented history of interactions allows advisors to “pick up where the last person left off,” as one 
advisor described it. 


The second benefit mentioned by advisors was having a record of advisors’ interactions with students to 
address miscommunications. An advisor told a story of a student expressing frustration that her financial 
aid would not apply to her courses. The advisor did not recall the interaction until she reviewed their 
interactions and realized the student had requested to change her major. When the advisor asked the 
student if they had submitted the change of major form she had provided, the student said no. The 
advisor was able to explain that was the reason why the students’ financial aid was not being applied to 
the courses she was enrolled in. Given the significant number of students BPCC staff often advise, a 
number of advisors mentioned the benefits of having interactions with students documented. 


The third benefit was the ability to target select groups of students for outreach or interventions based 
on filters chosen by advisors. For example, one advisor described using Inspire to identify students who 
had been admitted to BPCC but had not yet registered for Fall classes. Advisors pulled the list of 
students who fit this description and used Inspire to send students a personalized email encouraging 
them to register for courses. Overall, all front-line advising staff interviewed expressed satisfaction with 
the Inspire tool. 


Faculty Response to Intervention 
Initial Excitement with Mobile Apps 


The desktop version of Open Campus had been in existence for many years before BPCC began 
developing the mobile app versions. A few instructors reported that they were unfamiliar with the 
resource before the intervention began this semester, but the majority were aware of the tool. Both 
English and math instructors also reported incorporating Open Campus videos in their instruction in 
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various ways, with some even requiring students to watch the videos in order to receive points toward 
their grade. 


Despite the familiarity with Open Campus, the majority of instructors expressed genuine enthusiasm for 
the idea of providing students with mobile apps that allow them to access curricular content anytime, 
anywhere. This quotation from one instructor summarizes the general perception among the faculty: 


I think most of us, even the ones who didn't get chosen for [the treatment group], are interested 
in it, especially when we had the workshop. Everybody was very excited about it and that it was 
going to be available to the students. If the students can access something pretty easily, by just 
that app, they will go on it a lot quicker than having to go through a long list of things. I think the 
easiet it is for them to access, the more they'll use it. 


Benefits of the mobile apps were expressed even by instructors who were heavily reliant on the desktop 
version of Open Campus and were somewhat reluctant to abandon their current practice in favor of using 
the apps. One such instructor stated: 


I think the biggest difference that would [make me] want to really encourage the app is the 
chunking of the videos. Like you can go and watch the homework video, and it's like how long, 
and you've got to fast forward if you need. Whereas if they have the app, you can click that exact 
objective and spend two to three minutes learning about that objective, the only one you needed. 


In short, even though aspects of the implementation of the apps posed some challenges for faculty, the 
consensus seemed to be that instructors were enthusiastic about the prospects of providing students with 
mobile apps and using them in their instruction. 


Confusion Over Website Version of Mobile App 


Whereas there was genuine excitement expressed about the mobile apps, there was also confusion 
amongst faculty and students over the format of the apps. As discussed above, the first phase of the apps 
consisted of developing a new website for Open Campus built using a responsive design compatible with 
mobile devices. Although installing a shortcut to the site on one’s phone resulted in the site resembling 
and functioning as an app, the non-traditional installation process was a source of consternation for 
some. As one instructor described: 


The first time, when they talked about how they were developing this app, I was going into the 
App Store, and I was like, “I've been looking for this thing. Where is itr”? And in my instructional 
meeting they were like, “Oh it’s not an app yet. No, it’s just in development. It’s just through a 
website.” 


Even when faculty members handed out the instructions for accessing and installing the mobile apps 
created by BPCC staff, students were also confused about the version of Open Campus they were 
supposed to use. 


One instructor stated that many of her students had reporting using the app, but when she received her 
student engagement report it did not show these students as having ever logged into the system. She 
eventually learned that students had searched for BPCC’s Open Campus on the internet after misplacing 
the handout and had likely been using the older desktop version of Open Campus. 


Although students may have benefited from the resource, there was no way for the instructor to track 
her students’ engagement with the platform. Some students found the correct site, but only accessed it 
through a computer and never installed in on their phones. 
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The results of the interviews and focus groups suggest confusion will likely be reduced, and perhaps use 
of the apps increased, once they become available through the App Store or Google Play. As one 
instructor argued: 


I think once it’s an actual mobile application they can download in the mobile app store, I think 
that's different than ... I see something and I'm like, “Oh it's just a website, I'm saving it as a thing 
on my phone.” To me that’s not the same as a mobile application...I think once it's a mobile 
application everyone can download, I think that more [students] will be more inclined to use it. 


Randomized Controlled Trial Process 


A critical component of BPCC’s evaluation is the process of randomly assigning course sections to 
receive access to the mobile apps or not. Although this methodological approach is designed to produce 
rigorous evidence of the impact of the intervention, the process was unfamiliar and challenging for 
faculty. There were three key concerns related to the evaluation design that emerged during 
conversations with faculty. The first was being selected for the control group and not being able to use 
the apps at all. Many instructors were excited about the possibility of incorporating the mobile apps into 
their instruction when they heard that BPCC had been awarded a grant from the FITW program to 
implement the apps. One math instructor stated: 


I was real excited...they told us we would get an email letting us know if we were going to be 
chosen or not, so I was like, “God, I hope I'm chosen.” I knew I taught two classes, so I said, 
"Well at least maybe one of my classes will be chosen," because I was real excited as soon as I got 
back to the school with the information, I started going through it and taking quizzes and 
watching videos. Oh yeah. 


Although this instructor was selected for the treatment group, instructors who were not given access to 
the apps expressed frustrations. Some faculty who helped to produce the new video lectures for the 
mobile apps were selected for the control group. Although instructors were disappointed, they did 
express understanding of the reasons behind the evaluation design. The role BPCC staff played in 
communicating the purpose of the evaluation was key in maintaining buy-in among faculty, particularly 
those in the control group. 


The second concern was the timing of the course randomization. In order to minimize incidences of 
attrition and contamination (i.e. students assigned to the control group changing courses and getting 
access to the mobile apps, or vice versa), the third-party evaluator recommended that course sections be 
randomized to the treatment and control groups after the last day for students to modify their course 
registrations during the Spring 2017 semester (January 20, 2017). 


This meant instructors were not notified of which group they were in until roughly two weeks after 
courses had begun. Some instructors indicated that their course schedules were highly structured, making 
it difficult to modify the schedule in order to explain the apps during class time and incorporate them 
into instruction. For example, when asked about students’ use of the mobile apps, one instructor stated, 
“T don’t have any assignments geared toward they have to go on there because I had already designed my 
class prior to [being assigned to the treatment group].” Another instructor reported: 


Once we start teaching, we don't have time to stop a class and for 30 minutes take a trip to the 
[computer lab]...I mean and so we introduced it, what, two or three weeks into the semester? 
Well we’re on a roll and we have notes to get done. We told them about it, we handed the 
handout, we encouraged it but, I'm with you. I think it would have been different if we on day 
one, as part of our housekeeping stuff, could have all done it together. That way everybody knew 
how to do it. 
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Throughout the Spring 2017 site visit, multiple faculty members reported that more advanced notice of 
which group their course sections were in would have increased their implementation of the apps. As 
one instructor stated, “By the time we got the app going and everything, the semester was way in before I 
even found out that I was in [the treatment group].” 


This feedback from instructors contributed to the decision to randomize course sections earlier in the 
semester for the Fall 2017 intervention. Faculty reported being much mote satisfied with the timing of 
the randomization and when they were notified of their assignment to the treatment or control group for 
the Fall 2017 semester. One instructor summarized how more advanced noticed of the assignments 
contributed to her ability to more effectively integrate the apps into her instruction: 


Yeah, so I think last semester what happened was that we didn't know...which classes of ours, or 
if at all, our classes were going to be chosen to do this. I had already made my lesson plans, and I 
stick to my...I'm pretty structured with how I have every day planned out, so it was just hard for 
me to like work stuff in like that after I had already made the plan, because I think it was like two 
or three weeks into the semester maybe. So this semester we knew ahead of time, and so I 
actually had three or four different days scheduled where I was working in some of that material 
as either supplement or just part of the lesson. That helped me out as just a logistical thing. I 
probably altered five or six of the lessons that I would normally have taught. 


The third concern was confusion over the role of instructors in encouraging students to use the apps. 
Although some faculty members in the treatment group may have simply elected not to use the apps, 
others expressed confusion over the extent to which instructors should emphasize use of the apps. Three 
different instructors expressed similar confusion: 


Well, for me, all I did that first week was show it to them. I passed out the handout, this handout, 
which I thought was really well done. I thought, and I didn't have any students coming to me 
complaining that they couldn't follow the handout. Then I actually put it up on the projector and 
walked them through logging in and then I showed them what the class looked like and they all 
acted interested, but then I didn't really push it again. Because my plan was to use it to reinforce 
skills before exams. 


I guess maybe I was a little confused about the nature of the study. I kind of assumed it was just a 
monitoring to see what they did with it. I wasn't promoting it so much... Yeah because when we 
were presented the app, it was, “You don't need to change anything.” I got the impression that it 
was, I mean it is voluntary obviously but to just leave it alone. Maybe that's what y’all were 
measuring. I didn't know. 


I think that we’ve got to figure out a way to incorporate it into our course calendar somehow to 
get [students] involved so that we see that it actually works, because with the way it’s set up now, 
we don’t have to change what we had already planned, so therefore, here it is. It’s available to you 
but it's up to you whether you go on it or not. 


The lack of advance notice regarding which course sections would be granted access to the apps and 
confusion over the extent to which faculty were supposed to encourage use of the apps both contributed 
to relatively limited implementation early in the semester. One instructor candidly stated: 


As far as working it into my class as supplemental material, I haven't done that. I have failed to 
do that. I do plan to use it more. I just haven’t. 


Alignment with Pre-Existing Courses 
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One potential obstacle to instructors’ ability to seamlessly integrate the mobile apps into their instruction 
was the degree of alignment between the course content available through the apps and instructors’ pre- 
existing course schedules and syllabi. Despite this potential challenge, faculty members expressed high 
levels of satisfaction with the alignment between the content in the apps and the subjects they normally 
covered in the courses. When asked about how easy or difficult it was to integrate the apps into 
instruction, one faculty member stated: 


It was easy because everything they teach in the course is everything we're already doing. My 
lesson plans were laid out. I know exactly what was going to happen every day for the length of 
the semester, so I could just use the Open Campus as supplemental stuff for the students’ use. It 
was easy to throw in. 


This conversation between a different instructor and the evaluator surfaced similar reactions to the 
alignment with the instructor’s course schedule: 


Evaluator: Did you feel like there was alignment between your lesson plans and your outline 
for the semester and what- 


Instructor: Oh yes. 
Evaluator: Yeah? 
Instructor: Yes, very much so. Very much so. I was really excited because that's why I went 


on teal quick to see, does this line up with the things that I'm teaching? Man, it 
was bam, bam, bam. 


Overall, no instructors reported that the topics they covered in their classes differed in any fundamental 
way with the topics included in the app modules. However, at times there were slight pedagogical 
differences, such as instructors in the video lectures explaining a concept one way and in-person 
instructors explaining the same concept in a different manner. Faculty members had mixed responses to 
this situation. Some felt that explaining things differently would increase the likelihood that students 
would understand the material, given that students think through the same problems using differing 
strategies: 


Well, she touches on different ways to do things, too, I think, in some instances, because there's 
some things that you can do more than one way. It seemed to me that she touched on that. Being 
a math teacher, I'm not one of those that says, “You must do it this way.” I allow, there's 
numerous ways and kids think different. One sees it this way, and one sees it this way. You don't 
want to make him do the problem a certain way. If he totally gets it another way, don't put him in 
a box. 


However, other instructors expressed concern that explaining things too differently could become a 
source of confusion, particularly given that students enrolled in developmental education courses often 
struggle academically: 


The only thing that I think is a little bit different is some of the videos, teachers will explain 
things in a way that I don't teach it, or another math teacher might show things a different 
way...I'm sure other teachers do other things, so I think that could be confusing for a student 
who's not strong in math, if they're getting instruction from a teacher one way, and then they're 
going and watching a video and it's teaching them a different way. That's not extremely helpful. I 
try to steer clear, and don't assign those videos when the teacher is doing something different 
than me. 
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Without complete uniformity in pedagogical approaches among instructors, it is inevitable that some 
differences will exist between how an in-person instructor teaches a topic and how it is explained in the 
pre-recorded video lectures available through Open Campus. Although this is neither inherently positive or 
negative, feedback from faculty suggest it would be appropriate to discuss how to effectively account for 
these different approaches when instructors are being trained on incorporating the apps into their 
instruction. 


MyMathLab 


One of most significant obstacles to the implementation of the mobile apps in the developmental math 
courses was BPCC’s use of separate courseware technology called MyMathLab, a product of Pearson®. 
All math instructors at BPCC use MyMazbLab in developmental courses to assign students homework 
and have them complete activities. At the request of math instructors, BPCC had integrated the Open 
Campus videos into the MyMatbhLab portal. In the two developmental math course numbers that were part 
of the intervention (Math 98 and 99), students were required to at least click on the videos through 
MyMathLab in order to receive points toward their grade. One instructor described the process for 
students: 


When they go to open up an assignment for that day’s lesson, the video, it’s listed as a homework 
problem. They have to click on the video to get a point for that homework problem. They don't 
have to watch it, but it just was the easiest way for us to put it into the assignments to where they 
didn’t have to hunt for, you know, “I didn't understand the lesson today, where can I find that 
video?” They go to that homework, there’s a video if they need it. If they don't need it, we tell 
them, “Just click on it and click off, but that way you know where it’s at.” 


Because math instructors were committed to using MyMazbLab in their instruction, many felt it to be 
counterintuitive to request that students watch videos through the mobile app instead of MyMathLab. As 
one math instructor stated, “The videos are already in their homework, so why hunt for them when you 
can just click on it” 


During the focus group with math faculty, some instructors in the Spring 2017 site visit suggested that 
the Open Campus videos should be removed from MyMathLab for both treatment and control groups. The 
control group would still be able to access the Open Campus videos through the desktop site, and the 
treatment group would get access to the mobile apps. 


It was expected that this would encourage students in the treatment group to use the mobile apps 
without fundamentally changing the educational experience of students in the “business as usual” 
condition. This change was implemented in the Fall 2017 semester. Although some math instructors 
were displeased with this decision, most recognized that this approach would allow BPCC to collect 
stronger evidence of the efficacy of the mobile apps on student performance. 


Technical Difficulties with Mobile Apps 


In general, the response from faculty to the core components of the mobile apps was quite positive. 
However, faculty members did report a number of technical difficulties using the mobile apps. The 
following conversation between the third-party evaluator and a math instructor highlights these 
challenges. 


Evaluator: So how is it going so far? The implementation of the mobile apps? 
Instructor: Just okay. 
Evaluator: Just okay? 
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Instructor: Yeah. There were a lot of issues with them submitting quizzes, and it wasn't 
giving scores back. 


Evaluator: Is that why they’re having to take screenshots? 
Instructor: No. They have to take screenshots because I don't have access to their stuff. 
Evaluator: Oh, really? 


Instructor: Yeah. I don't get to see what they do unless they physically show me what they 
did. They either have to show me in person, live, or take a screenshot and send it 
to me, so that’s that, and there’s glitches in the system. It’s new, so that's going to 
be normal, but all the videos, I think were fine. It was just taking the quizzes and 
stuff. 


These concerns were echoed by other faculty members in both the Spring 2017 and Fall 2017 semesters. 
Instructors also reported that response options for quiz questions were incorrect. For example, different 
response choices would have the same answer (e.g. A = 40, B = 42, C = 42, etc.) or the question stem 
would ask for the value of X but the response options would contain values for Y. Although relatively 
minor, these issues were bothersome enough to be mentioned by instructors in multiple interviews and 
focus groups. These concerns have been relayed to BPCC staff designing the apps and have been 
addressed as they have been identified. 


Lack of Oversight Functionality 


In the Phase 1 version of the apps, faculty were not able to log in to the system and track their students’ 
progress, nor did they receive any type of automatic notification when students completed tasks within 
the app. This proved a challenge for faculty who required students to engage with the apps in specific 
ways. For example, some faculty required students to watch videos through the app. The only way 
students were able to submit evidence that they had completed this task is by taking a screenshot with 
their phone of the completion window that popped up once they had finished viewing the video. 


Faculty reported that BPCC staff overseeing the implementation of the apps would send them weekly 
reports with some information on students’ engagement with the apps, such as which students had 
logged in, the last time they logged in, and the number of videos they had watched. However, key pieces 
of information, such as which specific videos students had viewed and their grades on quizzes, were not 
included in these reports. Some faculty reported not recetving this information until about one month 
into the semester. For faculty teaching eight-week courses, this provided limited time for them respond 
to the information about student engagement. Other instructors reported receiving the reports but not 
reviewing them. 


For the apps to be even more useful, faculty requested that the apps include an electronic gradebook that 
would allow the faculty to assign specific videos and quizzes to students and factor only these assignment 
tasks into grading, assign weights to videos and quizzes in according with their grading policy, and 
automatically calculate student grades so that faculty would not have to do manual calculations. These 
features are planned to be incorporated into Phase 2 of the apps that will be released in Spring 2019. 


Continued Variation in Instructors’ Use of the Apps 


Instructors had limited time to integrate the mobile apps into their courses for the Spring 2017 semester, 
both due to their unfamiliarity with the apps and the timing of the randomization. At least some 
instructors were able to more fully integrate the apps into their courses during the Fall 2017 intervention. 
Multiple instructors mentioned requiring students to complete quizzes and watch videos within the apps 
as part of their homework. One instructor required students to take the quiz but only graded them on 
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their completion of the quiz rather than the percentage they answered correctly. “I made [the quizzes] 
worth points for their homework assignments, regardless of whether they got them right or wrong. It 
didn't matter,” she stated. “But what I did notice was that they were ... | had more students, this time, 
repeat the test to try to get it right, as opposed to just blowing it off. I like that.” Another instructor 
provided a similar account. 


I'm requiring them to take the test, at the end of the module. I've given them dates that they're 
due. I've coordinated that with what I'm teaching, obviously. It's worked out well. I've asked 
them specifically, "Did this help your Do you feel as though you came to class better prepared 
and sort of knowing what I'm going to dor" Several of them said, "Yes." They did. So, I was glad 
to hear that. 


Other instructors had students complete the quizzes in class as a way to practice concepts that the 
instructor recently covered through lecturing. Rather than a traditional quiz, where students only get to 
attempt it once and are graded for the percentage they get correct, this instructor required them to take 
the quiz repeatedly until they recetved a perfect score. Importantly, he did not use this approach in the 
Spring 2017 semester but did so in the Fall 2017 semester, reflecting his continued experimentation with 
how best to use the apps. He described the process in the following manner: 


I had been using it in my class really as like bonus opportunity [in the Spring 2017 semester]. I 
would make it a suggestion, but I didn't necessarily incorporate into the class. But, I still had 
some pretty good results at the end of the semester, especially in the students who I could go 
back and see had used it, their grades were demonstrably higher. So that was encouraging. 


So this time, I brought the pilot class to the writing lab, which is our computer lab on the second 
floor, and had them all log into Open Campus and I had them do the quizzes in two of the 
modules that corresponded with the midterm and I told them to take them until they made a 
hundred. Watching them, they got into it. I could see some of them wete getting into it almost 
like it was a game, which that is what we want. Cause if they’d score a 90, it’s like, "Oh! I got a 90, 
it’s so close!" And then they'd just dive right back in there, do it again. 


Yet other instructors used the apps quite differently, and at times in ways that may have been 
counterproductive. A key principle of the apps is that providing students with modularized videos allows 
them to re-watch concepts that they may have struggled with in class and to do so at their own pace. 
This “flipped classroom” approach would therefore consist of students watching videos on a topic either 
before or after that topic was covered in class. In contrast, at least one instructor mentioned instructing 
students to watch the videos during class time. Expectedly, he encountered difficulties with this 
approach. “Sometimes it’s not easy to incorporate [the apps] into the class,” he stated, “only because 
you're watching videos on your phone, so having 25+ people watching separate videos just doesn't 
work.” 


The majority of instructors reported that requiring students to use the apps in some manner, whether by 
assigning videos for homework or making students complete the quizzes, was more effective than simply 
mentioning the apps as a supplemental resource. However, faculty also reported that some students 
either had technical difficulties using the apps or did not have a smart phone, which would prevent 
students from using the apps altogether. For this reason some instructors indicated they were wary to 
make use of the apps a required part of the course, as that could unfairly disadvantage students for whom 
accessing the apps was a challenge. 


Faculty Use of the Apps to Improve Their Own Instruction 
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Although the apps were not specifically designed for this purpose, instructors in both the Spring 2017 
and Fall 2017 semesters reported watching the videos in the apps to see how their colleagues teach 
specific concepts in order to improve their own instruction. One instructor gave this account: 


As a matter of fact, I go on [the apps] and watch. I try to. I try to watch the videos...I did some 
of them, on the plane on the way home [from a trip]. I try to follow [the instructor in the video’s] 
lead...so that it would be consistent, because we’re teaching the same thing. So, there would be 
consistency for the student. That does matter. 


Instructors’ Positive Reactions to IDP 


Despite some ambivalence, many instructors recognized the value of a tool such as Inspire. One 
instructor described the course registration process as “complicated for students and for faculty. It's 
chaotic.” Any tool, including Inspire, viewed as potentially alleviating that confusion was welcomed. 
Inspire was viewed by some as particularly important given that both faculty and advisors are tasked with 
assisting students with course registration and students receive advice from many other departments and 
divisions on campus. As one instructor stated: 


If [a student is] in your particular division, you're talking with your advisor for that particular 
major. Then you come over here to advising and you talk to a different advisor, then we can see 
what’s going on in relationship between that interaction that you had with person A and the 
interaction I should be having with you versus me starting just totally from a blank state, even 
though you've seen three different people. That's the best thing because now I've had, even when 
we're doing some of the training, I've had it where some instructors say, well that's what I do. I 
actually write down information when I advise my students and stuff. 


I say, that’s great! That is wonderful. You should do that, but if you're the only one that is 
advising that student then that’s going to work out perfect. But if somebody else advises that 
student, they don’t have access to that information that you have. Now we have a device where 
you can put that information about what took place with that student as far as advising that 
student on his academic progress. And everyone has that information versus I have it here and 
that student is only going to come to me and get advice. That's just not realistic. It's not. 


Some instructors similarly felt that having a single source of information about students’ interactions 
with various departments, offices, and personnel across the college was extremely useful. As one 
instructor stated: 


I like the notes of Inspire, because as a teacher, I can't tell you how many times, after we've been 
into the semester, I have to figure out, “How did you get in to this class? You're not even... You 
haven't even met the prerequisites. Who put you in here?” And with that, being able to leave a 
note, “So and so got special permission from this instructor to be put in the class, even though it 
was closed,” because sometimes that happens. But there’s nothing for us to look at to see how 
this situation happened. So I like the note side of [Inspire]. 


Perhaps most importantly, instructors felt that Inspire was a tool that could empower faculty, help them 
more effectively serve their students, and ensure their advice to students was heeded by others on 
campus: 


We need to know more. The left hand needs to know more what the right hand is doing. I'll put 
it that way. If I talk to somebody then another faculty member comes and visits with that student 
six months from now, I don't want the things that I said ignored. I don’t. Especially, and a matter 
of fact I'm very adamant about it, if it’s a student [in my program], I really want to talk to that 
student myself or somebody in the program. That cannot always be done, especially these new 
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students. I want our students to have very directed advising...I want others to know it’s serious 
to me. My job is serious. It’s more than just advising. 


Instructors’ Negative Reactions to IDP 


There were two factors that tended to result in BPCC faculty reacting negatively to the tool, at times 
quite vehemently. The first was the perception that reforms tend to come and go at BPCC, and possibly 
in higher education more generally, that made people skeptical of any new reform that requires time and 
effort from instructors. One faculty member described his initial response to hearing about the 
purported benefits of Inspire: “You get skeptical after twenty years and everything so you're like okay, 
yeah, right.” 


The other factor causing resistance to Inspire from faculty was a general resentment toward being 
required to advise students. A majority of instructors interviewed during the site visit felt that the 
requirement from BPCC administration for instructors to advise students during specific registration 
times during the semester detracted from their teaching, was outside of their proper role as instructors, 
and was something they were not properly compensated for. This sentiment colored instructors’ 
perceptions of Inspire. As one instructor described her reaction to Inspire: 


I don’t care. I don’t...I hate to advise. I'm a teacher. I'm a therapist. I don’t want to advise. I 
think it’s probably a great plan. It’s a great project. It’s a great this. It’s a great that. It’s just 
another annoyance to faculty...I doubt any of us will do it. If we do, it’s going to be 
begrudgingly, because we don't have the time to pull this up, and pull that up, and do this, and do 
that, and all these fabulous ideas are great, but nobody's making any more time for us. Nor is 
anyone offering us any more money, in eleven years. So, let's just throw that in there...We 
cannot do one mote free thing...[{Inspire] is a thorn in my side. 


A different instructor shared similar sentiments: 


I'm not a fan [of Inspire] at all. But it has less to do with it and more to do with we don’t want to 
do that. And I went to the introduction thing, I said, “Oh, this is great. It’s more work for me.” 
And that's no problem other than that is in my contract, I get that, but I still don't like it. And 
[Inspire] is...it’s more entailed. And I won’t do it. So, there you go. 


Even when instructors were not as vehemently opposed to the use of Inspire as the instructors quoted 
above, they often felt that the tool would be of limited value given how infrequently they are required to 
advise: 


In the advising center, they do this all semester long. I do it one week out of the semester. And 
for the first couple of students, I really, I have to relearn how to do it over again. And that's just 
using, you know, the other stuff [such as the course registration system]. I just don't it often 
enough to feel like I have a mastery of it. And so I worry that the first time I go into Inspire... 
I'm going to have to learn it all over again. 


Similarly, whereas instructors felt capable of advising students about the required courses to take in 
majors in the department where the instructor taught, students would often come with far more 
complicated questions, particularly those related to the transferability of courses. One instructor 
mentioned advising students with these questions, and the interviewer asked her if she felt she should be 
providing this type of advising. The instructor responded: 


In theory, no, I should not be. And I told [the student], I was like, “I can only put you in 
stuff... If you think this is going to transfer, then you need to go see an actual advisor in the 
advising center and have them do a degtee audit," because I didn't even know where to start with 
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that. I would feel confident making a math decision, but even then, these [courses] were from ten 
years ago, and how do I find her transcript from ten years and then the course description from 
ten years ago? That to me, that’s too much for somebody who is not an advisor. I'm a math 
teacher who advises on the side. 


Given some instructors’ negative perceptions to advising overall, it will likely require a significant shift in 
these instructors’ thinking for them to support BPCC’s adoption of Inspire and use it themselves. As one 
instructor advocating for Inspire stated, “We have to get into the mindset that it’s not something that’s a 
burden on us, but it’s something that’s there to help alleviate a lot of the issues that we have.” 


Advisors were similarly concerned about faculty members’ unwillingness to use Inspire, as that would 
severely limit the usefulness of the tool and result in advisors having less information about students’ 
interactions with instructors. One advisor expressed these concerns while restraining her criticism of the 
faculty: 


We would have to convince the faculty to do input [in Inspire] or to have somebody that they 
could contact or send the students to as soon as the students start having...We have a wonderful 
faculty. There’s a lot of faculties [sic] that help the students tremendously, but as soon as they get 
into unique type of difficulty, it’s like any place else. We have faculty who are not as willing to 
work with students. They’re there to teach. That’s it. 


Mixed Reactions from Faculty to IDP 


Although Inspire had strong advocates and fierce critics among the faculty, the instructor interviews 
suggested that the most common reaction was mixed. Many faculty felt that Inspire could be a very 
valuable tool, particularly if other advisors on campus were diligent in documenting their interactions 
with students that faculty could then see in students’ profiles. However, the most significant challenge for 
faculty was the additional time it takes to input their notes from student advising sessions into Inspire. 
The following response from one faculty member was typical of this sentiment: 


My initial impressions would be, it would be a wonderful tool that could help us. My later 
impressions are I wish we had more time to use it to the best of its benefits...I think the day 
before yesterday maybe we saw a total of 20 people. Yesterday we saw 60 some odd. There gets 
to be a lot of students waiting outside. If a student is just a brand new student that needs new 
information, I do not turn to [Inspire]. Where I try to turn to it, and maybe I don't turn to it 
enough, is when students are having problems. So, if we have the time to do it...that’s been one 
holdup on my part, is finding the time to really do it the way it should be done. 


Instructors and advisors’ perception of having limited time to conduct advising was also exacerbated by 
the fact that both instructors and advisors reported using alternative methods for taking notes regarding 
advising sessions. 


One instructor mentioned recording interactions with students in a spreadsheet. Although he admitted 
that Inspire was far more beneficial in that he could see the students’ interactions with other people on 
campus and he did not have to go back through spreadsheets, his department had not yet abandoned the 
old practice of using spreadsheets to notate advising sessions. Until advisors and faculty members in 
similar situations decide to discontinue the use of old methods for documenting advising sessions, using 
Inspire will duplicate the work needed to record their notes. This may limit adoption of Inspire even in 
instances where instructors and advisors recognize the benefits of Inspire. 


Non-Implementation of Components of IDP 
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Although BPCC experienced some challenges in generating buy-in for Inspire, particularly among faculty, 
implementation of the tool has been largely successful—the tool itself is fully functional, the majority of 
instructors and staff had received training on Inspire by the end of the third grant year, and use of the 
tool was growing rapidly across BPCC. Inspire is the only component of the IDP that was planned to be 
supported by grant funds. However, BPCC intended to implement two other components of the IDP 
created by Civitas, the Illume and Illume Impact tools. 


The base Hlume package integrates data from multiple sources together and uses algorithms to identify 
student experiences most related to student outcomes, and the Illume Impact tool allows users to 
estimate the impact of an intervention by matching students who received an intervention to a control 
group. The long-term vision is for BPCC to integrate data from the mobile apps into the IDP in order to 
assess how students’ use of the mobile apps influences their persistence and attainment and study the 
effect of developmental education interventions on student outcomes. 


A number of technical challenges have plagued the implementation of Illume and Illume Impact. The 
most notable was the Louisiana Community and Technical College System’s (LCTCS) requirement for all 
community colleges in the system to use the Banner learning management system. BPCC’s transition to 
Banner seems to have impeded their ability to prioritize implementation of Illume and IHlume Impact. 
Similarly, Civitas represented to BPCC that connecting Illume to BPCC’s data systems would be 
relatively straightforward, yet Civitas also appears to have underestimated the complexity of the task. 


These technical difficulties resulted in discrepancies and inaccuracies in the data reported in Illume, 
which subsequently sapped support for the tool. As one administrator noted: 


In almost all the trainings [on Illume] that we had, it came up almost every time that the data 
wasn’t being pulled accurately. A lot of people were saying, “Why would I use this if I'm not 
getting accurate information anyway?” So they just kind of quit using it or quit even looking at it 
because they thought it’s not pulling [data] accurately. 


BPCC staff instrumental in the FITW initiative continue to believe that the ability to integrate data from 
the mobile apps into the IDP would be extremely beneficial, particularly when considering how to 
evaluate developmental education interventions. However, [lume and I]lume Impact fall outside the 
scope of their purview given that grant funds were not used to cover the costs of those tools. It remains 
to be seen whether BPCC will be able to implement those components of the IDP in the final year of the 
grant or beyond. 


Student Use of Mobile Apps 
Initial Access to the Apps 


Due to the randomized controlled trial approach to the evaluation, BPCC had to ensure that only 
students who were enrolled in one of the course sections assigned to the treatment group at the time of 
randomization received access to the apps. To do this, BPCC pulled lists of students’ email addresses and 
sent them individualized emails with information about how to access and download the apps. 


Students were provided a login account and were instructed to reset their passwords the first time they 
logged into the system. In addition to these emails, instructors of course sections assigned to the 
treatment group were provided a handout with instructions that they distributed to students in class. 
Some faculty took time in class to assist students with logging in and downloading the apps, although it is 
unclear how pervasive this approach was across faculty. 


Some faculty members expressed concerns that students would not respond to the emails, or the number 
of steps required to download the app would discourage them from doing so: 
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Maybe if there’s some way that we could've actually signed the kids up instead of them having to 
go on and say, “I forgot my password,” and they had to create their own password. If there was a 
way that we knew their student ID and we could say, “Here’s the website, here’s your username, 
here’s your password.” With college kids, from the time I give them the paper, by the time they 
get home, they may not even still have that paper that tells them how to do it. 


This theme also emerged in student focus groups. 


Evaluator: Do you remember when you first heard about this specific app ... For the two of 
you that were in this class [in the treatment group]. Do you remember how it was 
described? What's the purpose that you're doing this? Do you remember? 


Student 1: I just seen the email. 

Evaluator: You don't remember hearing anything about it in class? You just got an email 
basically saying- 

Student 1: Yeah, I remember it, but I don't remember what he said. 

Student 2: I know there's a grant or something I got. 

Evaluator: ...Was this the first time that you basically had an opportunity to download an 
app [for a class]? What did you think? 

Student 2: Awesome. 

Evaluator: Yeah? You thought awesome? That was your first impression? 

Student 2: Yeah, I liked that. 

Evaluator: [gesturing to Student 1] What about your 


Student 1: I didn't really think nothing of it. 
Evaluator: Yeah? 
Student 1: Because I didn't really open it. 


The Spring 2017 site visit was conducted relatively early in the semester, and at a time when there was 
still confusion among the faculty about how extensively they should incorporate the apps into their 
instruction. For these reasons it is not entirely surprising that some students did not fully embrace the 
apps immediately after being granted access to them during the Spring 2017 intervention. 


Some instructors tried different approaches in the Fall 2017 semester. One instructor recounted 
approaches that her colleagues had used: 


I think in some of the classes that have the computer labs, I think they had the whole class log in 
for the first time together. And I think, once they had an account, and it was there, they would 
probably be more apt to use it. But in a regular classroom where I just told them about it and I 
didn't make it a requirement, I don't think they have the motivation to go do that on their own. 


Engagement with the Mobile Apps 


Student engagement with the mobile apps was limited early in both the Spring 2017 and Fall 2017 
semesters. Multiple instructors reported that they had no students logging into the system at the time the 
first student engagement report was disseminated to faculty in the spring, even when they walked 
students through the login process in-class. Indeed, low student engagement was a primary reason the 
Spring 2017 site visit originally scheduled for the end of the semester was conducted in February instead. 
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As discussed above, many faculty members did not emphasize the apps much early in the semester, and 
if they did they used the app courseware as a supplementary resource rather than as an integral part of 
work for the class. This is a likely cause of low student engagement with the apps early in the semester. 
As one instructor stated: 


Most of our students, if they don't see some kind of reward, most of them aren't going to [make 
the effort]. You're going to have the overachievers who, if it's going to benefit them any kind of 
way, they will go on there, but the other ones are waiting for, “What do I get if I go on there and 
do this?” They don't think as, “How's it going benefit me? How is it going help me prepare? 
How's it going help me to become a better writer?” and so on. Most of them don't have that 
mindset. You have to kind of push them...some of the other ones, you have to kind of nudge 
them, so I think we've got to figure out a way to get them in there to actually use it. 


After the site visit in late February 2017, it appears that instructors made a more concerted effort to 
emphasize use of the apps with students and engagement with the apps increased. However, engagement 
declined again towards the end of the Spring 2017 semester, despite the utility of the apps for allowing 
students to improve in areas where they struggled during the semester and prepare for their final exams. 
These patterns were similar for the Fall 2017 cohort. Overall, about one-fifth of students assigned to the 
treatment group in both the Spring 2017 and Fall 2017 semesters accessed the mobile apps and watched 
at least one video, although a higher percentage of students utilized some version of Open Campus if the 
students who accidentally used the older desktop version are included. 


Although students did report technical glitches with the apps, even students who expressed concerns 
often engaged with the apps in desirable ways. One student gave this description when asked about how 


she used the app: 


Student: I go on there, I look at the video and I'm like, “Oh, okay. That makes sense,” and 
then some of the questions I was like, “I don't understand this.” 

Evaluator: For the quizzes you're saying? 

Student: Yeah. 

Evaluator: You like the videos, but the quiz is still...you weren't totally sure? 

Student: Yeah, it took me forever to finally get a good grade. Retry, retry, retry. 

Evaluator: Did that process help you figure out which ones were wrong or not? Like taking 
the quizzes multiple times? 

Student: Yeah. 

Evaluator: Do you feel like that helped you retain the information or were you just like, “I'm 


going to just keep pressing options until I get the grade,” but you're not really 
retaining the information? 


Student: I was working it out. I would look at the problem like, “Which one did I miss.” 


Sometimes they wouldn't even show up on the same quiz. I was like, “Okay, I see 
how I do this.” 


Overall, students who engaged with the apps in any substantive manner reported high levels of 
satisfaction with their functionality and engaged with the apps in ways that literature suggests will lead to 
improved retention of information and performance in courses. The key concern for BPCC moving 
forward is strategizing ways to increase student engagement with the apps and addressing any technical 
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issues that arise which could lead to students being turned off by the apps before they realize the benefits 
of using them. 


Characteristics of Students Who Use and Do Not Use the Apps 


The mobile apps are designed for all students enrolled in one of the targeted developmental education 
courses, but instructors described how student characteristics might influence whether or not students 
choose to use the apps. On one hand, students who were performing very well in the course may be 
disinclined to invest additional time in a supplemental resource like the mobile apps, particularly if use of 
the apps was not a required component of the course. This concern was discussed by one instructor: 


I have a group of 26 in here and I happen to have a very good smart group that obviously has 
had algebra 2 before so they're just refreshing themselves. So my first two quizzes I had a lot of 
A's and a lot of B's so hence I don't have very many feeling the need to get on the app for 
remediation. 


On the other hand, multiple instructors reported finding that students who needed support the most 
were often the least likely to use the apps. As one instructor stated: 


Some have said so far that the issue with making it optional is that the students who need it the 
least are the most likely to do it, and the students who need it the most are the least likely to do it 
sometimes. If you make something optional then the kids who you really are like, “They could 
benefit from this,” are the ones that aren’t going to do it. So it’s that challenge of creating that 
incentive. 


Supplemental analyses not contained in this APR suggested that lower performing students were in fact 
more likely to use the apps among the sample of students assigned to the treatment group. However, it 
may be the case that lower achieving students are more likely to use the apps compared to very high 
achieving students, but very low achieving students may be the least motivated to use the apps. 


Student Perceptions of the Mobile Apps 


Among students who used the apps, perceptions of their functionality and ease of use were mixed. Some 
students reported no technical issues and tremendous benefits from using the apps. For example, one 
student who was interviewed recalled having no problem installing the app on her phone, felt that the 
mobile app courseware was perfectly aligned with what she covered in class, and had completed the 
majority of quizzes and videos in both of the developmental classes in which she was enrolled. This 
student described the app as an excellent resource: “It's pretty cool. The app is very great though. It is.” 


One of the key differences between the previous version of Open Campus and the mobile app versions 
was the modularization of the video lectures. Students reported high levels of satisfaction with the length 
and quality of the videos. However, students also reported that they did not initially realize that the 
quizzes covered entite modules whereas the videos covered sub-sections within the modules. For 
example, a particular module may have six or seven sub-sections with a different video on each, but the 
quiz would cover all of the sub-sections. Students began taking quizzes before watching all of the videos 
in a module, leading to them encountering questions for which they were not sufficiently prepared. 
However, students appeared to learn this relatively quickly and adjust their strategies for engaging with 
the apps accordingly. 


Unfortunately, the technical difficulties students experienced with the apps may have hindered their 
engagement with them. This conversation with one instructor highlighted how students responded to 
these technical difficulties. 
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Evaluator: Have you gotten any student feedback from their use of the apps? 


Instructor: Yeah. They don't like it. They laughed about it because they said, “This quiz won't 
submit. I don't know what your problem is.” Or, there would be a problem that 
the variable is y, but all the answer were x's, which was not right. There would be 
answers that were the same, so that was a little bit confusing for them. Is it B or is 
it D? It's the same, x to the nth. Just little things like that. They didn't like that. 
They need it to work smoothly, and so they need it to be concrete, and really 
correct, or else it just throws them off, or they just laugh about it and move on, or 
they do that kind of thing. It's just been okay, as far as the quizzes go. The videos 
are fine. Videos are great. 


Evaluator: If the apps were completely optional, where you didn't require them to watch the 
videos on the apps, or do anything on the apps, how much do you think they'd be 
using the app right now? 


Instructor: Zeto. 


Although students did encounter some issues in using the apps, the overall student response to the apps 
was positive. In a student survey administered to students in the treatment group classrooms at the end 
of the semester, the vast majority of students said they were somewhat satisfied or very satisfied with the 
apps and agreed or strongly agreed with the statement: “Using the mobile app helped me learn the 
material covered in the class.” Additionally, more than 90% of students who used one of the apps and 
responded to the survey said they would recommend the apps to a friend. These findings are tentative, 
however, as the response rate to the survey was quite low. 


Faculty were sensitive to students’ perceptions of the apps. When asked what factors would most 
strongly influence instructors’ use of the apps in the classroom, numerous faculty mentioned students’ 
responses to the apps as being a primary determinant. Thus, buy-in from faculty and students are 
mutually reinforcing — students are less likely to use the apps if faculty don’t support them and integrate 
them into their teaching, and faculty are less likely to promote the apps if they feel they do not resonate 
with students. It will be important for BPCC to continue soliciting feedback from students to ensure they 
are reporting positive experiences with the apps. 


Greatest Barriers and Facilitators to Implementation 


The previous sections presented a more granular description of how implementation of the mobile apps 
and IDP through the first three years of the grant. The purpose of this section is to summarize what 
appear to be the greatest barriers and facilitators to implementation. 


Barriers 


The first barrier was confusion among faculty over the nature of the project and their role in its 
implementation. Although not all faculty shared this perspective, a significant portion of the instructors 
interviewed reported that they believed their function was to introduce the mobile apps at the beginning 
of the semester and remind students occasionally about their availability. Some believed that emphasizing 
the mobile apps too much could somehow bias the results of the evaluation, despite the fact that the 
efficacy of the apps could not be determined without students using them. It does appear that the site 
visit aided in clearing up misunderstandings among faculty about the nature of the evaluation, as student 
usage of the apps increased after that point. BPCC also conducted additional professional development 
for faculty before the Fall 2017 semester in order to support their implementation of the apps. 
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The second barrier was the technical difficulties encountered with the app and Inspire. Such issues are to 
be expected when new technology is being developed and implemented, but nevertheless both faculty 
and students reported frustration with these challenges. The most frequent issues related to the apps 
reported by stakeholders were the inability for students to show they had completed a task in the mobile 
app without taking a screenshot, quizzes not submitting correctly, and answer choices in the quizzes 
containing errors which made it challenging for students to select the correct answer. These issues were 
relayed to the BPCC FITW team after the first semester and are being addressed in phase two of the 
mobile apps. Additionally, some faculty were not accustomed to using technological tools, such as mobile 
apps or platforms such as Inspire, much at all. As one instructor stated of these instructors, “Other 
faculty...insist on having the paper, writing all their grades down, checking role on paper. I think 
especially those faculty that are still insisting on having the paper logs, that’s going to be where this 
would be an issue because they're not as adaptive to the changes.” This unfamiliarity with technology led 
some faculty to shun Inspire altogether before utilizing it in any significant amount. 


The third barrier was the timing of the implementation of the apps. Due to the decision to notify 
instructors of whether they had been granted access to the apps or not weeks into the semester, 
instructors felt that they would have to carve time out of their instruction in order to assist students in 
installing and utilizing the apps. Additionally, instructors had already finalized their syllabi at that point, 
making it difficult for them to modify their assignments or grade calculations after the apps had been 
implemented. These concerns raised by faculty led to the decision to notify instructors of their 
assignments earlier for the Fall 2017 intervention. 


The fourth and final barrier, or perhaps challenge, to implementation was the delicate balance BPCC 
staff overseeing the FITW project had to play in supporting implementation. On one hand, these staff 
were highly sensitive to the needs and challenges of instructors and did not want the FITW initiative to 
be viewed as a top-down mandate. On the other hand, both the mobile apps and Inspire could only be 
effective if instructors actually used them, and many instructors did not have much interest in doing so. 
Instructors themselves expressed competing views on the role that BPCC administrators should play in 
promoting implementation. Whereas some faculty members wanted to protect their autonomy in 
choosing whether to use specific resources recommended by the college, other instructors felt that use of 
the mobile apps and Inspire should be a requirement. BPCC staff overseeing FITW have used a variety 
of effective strategies to promote implementation without mandating use, but this tension between 
instructor autonomy and institutional mandate appears likely to persist. 


Facilitators 


The first facilitator was the support of lead faculty within both the English and math departments. FITW 
staff at BPCC recognized that faculty buy-in was critical to the successful implementation of the mobile 
apps and IDP, and the most effective strategy BPCC used to increase faculty buy-in was enlisting the 
input and support of lead faculty who were esteemed by their colleagues. These faculty were instrumental 
in designing the mobile apps, administering professional development to train other faculty in using the 
apps, and in incorporating the IDP into their practice. This resulted in faculty viewing the FITW 
initiative as faculty-driven rather than a top-down mandate from central administration. The continued 
support of lead faculty will be key to the future success of the initiative. 


The second facilitator was the materials BPCC staff distributed to students and faculty explaining the 
initiative and the in-class demonstrations of the product they conducted. Although FITW staff at BPCC 
had presented on the mobile apps in some classrooms early in the semester, when they realized that 
student use of the mobile apps was lower than anticipated they made a concerted effort to visit all of the 
classrooms assigned to the treatment group in order to explain to students how to effectively use the 
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apps and address question and concerns that had arisen. This in-person approach appears to have been 
related to the increase in up-take later in the semester. 


Finally, the last facilitator to implementation was the additional training and support of faculty as 
implementation proceeded. As discussed above, one of the most significant barriers that arose early in 
the semester was confusion over the nature of the evaluation and instructors’ role in implementing the 
apps. Multiple instructors expressed misunderstandings about the extent to which they should emphasize 
the use of the apps to students and incorporate the apps into their instruction. In close collaboration 
with the third-party evaluator, FITW staff devised strategies for clearing up these misconceptions and 
encouraging instructors to utilize the apps without making instructors perceive the initiative as a top- 
down mandate they were being forced to implement. In multiple instances, the very same instructors 
who expressed misunderstandings during the site visit and whose students had not engaged with the 
mobile apps at all by early February had extremely high rates of student engagement by the end of the 
semester. This likely would not have occurred without the efforts of FITW staff to address these 
misconceptions. 
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IMPACT EVALUATION RESULTS 


The impact evaluation for this fourth and final APR estimates the effect of the mobile apps on students’ 
short- (developmental education performance), medium- (next semester persistence and performance), 
and long-term (degree attainment) outcomes for both Cohort 1 (Spring 2017) and Cohort 2 (Fall 2017). 
Estimates for Cohort 1 alone ate found in the second APR, and estimates for Cohort 2 alone are found 
in the third APR. The research questions that guided the impact evaluation are as follows: 


1) What is the effect of access to mobile apps on the rates of students passing one of the three 

targeted developmental education courses (English 99, Math 98, and Math 99)? 
g. Does this effect vary by course number? 

2) What is the effect of access to mobile apps on the grades students receive in the three targeted 

developmental education courses (English 99, Math 98, and Math 99)? 
h. Does this effect vary by course number? 

3) What is the effect of access to mobile apps on the semester-to-semester persistence of students 
enrolled in one of the three targeted developmental education courses (English 99, Math 98, and 
Math 99)? 

i. Does this effect vary by course number? 

4) What is the effect of access to mobile apps on the next semester GPA of students enrolled in one 

of the three targeted developmental education courses (English 99, Math 98, and Math 99)? 
j. Does this effect vary by course number? 

5) What is the effect of access to mobile apps on the overall course passing rates during the next 
semester of students enrolled in one of the three targeted developmental education courses 
(English 99, Math 98, and Math 99)? 

k. Does this effect vary by course number? 

6) What is the effect of access to mobile apps on the credits earned during the next semester of 
students enrolled in one of the three targeted developmental education courses (English 99, Math 
98, and Math 99)? 

1. Does this effect vary by course number? 

7) What is the effect of access to mobile apps on the credential attainment rates of students who 
enrolled in one of the three targeted developmental education courses (English 99, Math 98, and 
Math 99)? 

a. Does this effect vary by course number? 


Developmental Education Course Pass Rates 


We begin by examining the rates at which the combined cohorts passed their developmental education 
courses. The results of the statistical model are found in Table 4. Across both courses and all course 
numbers, students in the treatment group were estimated to be 5.5 percentage points more likely to pass 
their developmental education course compared to students in the control group, a statistically significant 
difference (p = 0.009). 


Table 4: Treatment Model of Developmental Education Course Pass Rates for Combined Cohorts, no Interaction 


68 SE p-value 95% Conf. Int. 
Treatment Group 0.055 0.021 0.009 0.014 0.097 
Credits Attempted Pre-Intervention -0.014 0.001 0.000 = -0.017 -0.011 
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Credits Earned Pre-Intervention 0.017 0.002 0.000 0.013 0.021 
Race/Ethnicity (White) 
Black -0.102 0.031 0.001 -0.163 -0.041 
Other -0.060 0.024 0.014 -0.108 -0.012 
Female (Male) 0.102 0.022 0.000 0.059 0.146 
Pell Received 0.000 0.000 0.000 0.000 0.000 
Age Category (<18) 
18-21.99 -0.079 0.058 0.174 -0.193 0.035 
22-24.99 -0.049 0.066 0.460 -0.179 0.081 
25-29.99 0.039 0.065 0.546 -0.088 0.166 
30-39.99 0.030 0.064 0.643 -0.096 0.156 
40-49.99 -0.082 0.071 0.250 -0.221 0.057 
50+ -0.109 0.093 0.238 -0.291 0.072 


Block ID (Block 1) 


2 -0.001 0.097 0.993 -0.191 0.189 
3 0.055 0.103 0,595 -0.147 0.257 
4 -0.073 0.102 0.473 -0.274 0.127 
5 -0.217 0.099 0.028 -0.410 -0.023 
6 0.062 0.111 0.581 -0.157 0.280 
7 -0.009 0.099 0,925 -0.204 0.185 
8 -0.074 0.094 0.433 -0.258 0.111 
9 -0.131 0.099 0.184  -0.324 0.062 
10 -0.053 0.099 0.593 -0.247 0.141 
11 -0.183 0.099 0.066 -0.377 0.012 
12 -0.089 0.091 0.328 -0.268 0.089 
13 0.009 0.116 0.938 -0.218 0.236 
14 -0.037 0.097 0.700 -0.227 C152 
15 -0.062 0.095 0.513 -0.250 0.125 
16 0.024 0.086 0.783 -0.144 0.192 
17 0.141 0.114 0.218 -0.083 0.365 
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18 0.002 0.096 0.984 — -0.186 0.190 
12 -0.024 0.096 0.806 -0.212 0.165 
20 -0.126 0.096 0.192 -0.314 0.063 
21 -0.154 0.100 0.124  -0.351 0.042 
22 -0.167 0.090 0.063 -0.343 0.009 
23 -0.076 0.094 0.417 -0.261 0.108 
24 -0.098 0.094 0.297 -0.283 0.086 
25 -0.194 0.096 0.045 -0.383 -0.005 
26 -0.293 0.103 0.005 -0.496 -0.090 
2 -0.190 0.098 0.052 -0.382 0.002 
28 -0.145 0.086 0.093 -0.313 0.024 
29 -0.194 0.088 0.028 -0.367 -0.021 
30 -0.046 0.095 0.626 -0.231 0.139 
31 -0.153 0.082 0.062 -0.314 0.008 
Intercept 0.723 0.097 0.000 0.532 0.914 
var (Int) 0.000 0.001 0.000 = 112612.500 


Notes: Observations = 2,116, Course Sections = 80 


The next model found in Table 5 adds course number by treatment interaction terms to the model of 
developmental education course pass rates for the combined cohort. The largest estimated effect was for 
students in Math 99, in which treatment students were estimated to be 12.2 percentage points more likely 
to pass their developmental education course compared to control students, a statistically significant 
difference (p = .001). The estimated effects were smaller for English 99 (12.2 - 10.3 = 1.9) and Math 98 
(12.2 — 9.4 = 2.8), and both interaction effects were significant at the more liberal p < 0.10 threshold but 
not at the p < 0.05 level. The post-estimation test found suggestive evidence of significant variation in 
the treatment effect by course number (p = 0.093). 


Table 5: Treatment Model of Developmental Education Course Pass Rates for Combined Cohorts, Treatment by Course 
Number Interaction 


8 SE p-value 95% Conf. Int. 
Treatment Group (Math 99) 0122 0.037 0.001 0.049 0.195 
Course (Math 99) 
Engl 99 0.215 0.087 0.014 0.044 0.386 
Math 98 0.064 0.061 0.296 ~—--0.056~— 0.184 
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Treatment by Course 
Engl 99 -0.103 0.054 0.055 = -0.209 ~—-0.002 
Math 98 -0.094 0.050 0.061 -0.193 0.004 
Credits Attempted Pre-Intervention -0.014 0.001 0.000 -0.017 — -0.011 
Credits Earned Pre-Intervention 0.017 0.002 0.000 0.013 0.021 
Race/Ethnicity (White) 


Black 0.099 0.031 0.002 -0.161  -0.038 
Other 0.058 0.024 0.017 0.106  -0.010 
Female (Male) 0.103 0.022 0.000 0.059 0.146 
Pell Received 0.000 0.000 0.000 0,000 0.000 
Age Category (<18) 
18-21.99 0.078 0.058 0179 0.192 0.036 
22-24.99 0.046 0.066 0.491 -0.175 0.084 
25-29.99 0.044 0.065 0.498 -0.083 0.171 
30-39.99 0.035 0.064 0.583 -0.091 0.161 
40-49.99 0.077 0.071 0.276 0.216 0.062 
50+ 0.101 0.093 0.276 -0.282 0.081 


Block ID (Block 1) 


2 -0.003 0.096 0.977 -0.192 0.186 
I) 0.056 0.102 0.581 -0.144 0.257 
4 -0.076 0.102 0.457 -0.275 0.124 
) -0.219 0.098 0.025 -0.412 = -0.027 
6 0.062 0.111 0.574 -0.155 0.279 
7 0.133 0.078 0.087 -0.019 0.286 
8 0.070 0.071 0.328 -0.070 0.209 
9 0.012 0.077 0.873 -0.138 0.163 
10 0.091 0.078 0.240 -0.061 0.244 
11 -0.038 0.078 0.624 -0.190 0.114 
12 0.050 0.067 0.454 -0.081 0.182 
13 0.154 0.098 0.116 -0.038 0.346 
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14 0.127 0.069 0.066 -0.009 0.262 

15 0.099 0.068 0.143 ~—--0.034 0.232 
16 0191 0.053 0.000 0.087 0.295 
£7 0.318 0.094 0.001 0134 0.502 
18 0.000 0.095 1.000 -0.187 0.187 
19 0.025 0.095 0.791 0.212 0.162 
20 0.127 0.095 0183 0.314 0.060 
21 0.160 0.100 0109 0.355 0.035 
22 0.164 0.089 0.067 -0.339 0.011 
23 0.068 0.071 0.340 -0.071 0.206 
24 0.046 0.071 0.522 -0.094 0.185 
25 0.050 0.074 0.500 0.195 0,095 
26 0.157 0.083 0.059 0.320 0.006 
27 0.050 0.076 0513 0.199 0,099 
28 0.000 (omitted) 
29 0.022 0.057 0.708 -0.134 0.091 
30 0115 0.066 0.079 -0.013 0.244 
31 0.000 (omitted) 

Intercept 0.526 0.071 0.000 0.387 0.665 

Var(Int) 0.000 0.000 0.000 0,000 


Notes: Observations = 2,116, Course Sections = 80 
Developmental Education Course Grades 


The next analyses estimate the effect of access to the mobile apps on the grade students received in their 
developmental education courses for both cohorts. The first model estimates an overall treatment effect 
and does not include treatment by course interactions. The results of this model, found in Table 6, show 
that students in treatment course sections received a grade in their developmental education course 0.14 
points higher than students in the control group on average, a statistically significant difference (p = 
0.032). 


Table 6: Treatment Model of Developmental Education Course Grades for Combined Cohorts, no Interaction 


I) SE p-value 95% Conf. Int. 


Treatment Group 0.136 0.063 0.032 0.012 0.260 
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Credits Attempted Pre-Intervention —-0.045 
Credits Earned Pre-Intervention 0.049 
Race/Ethnicity (White) 
Black -0.330 
Other -0.143 
Female (Male) 0329 
Pell Received 0.000 
Age Category (<18) 
18-21.99 -0.311 
22-24.99 -0.174 
25-29.99 0.128 
30-39.99 0.227 
40-49.99 -0.209 
50+ -0.183 
Block ID (Block 1) 
2 -0.353 
) -0.163 
4 -0.403 
2) -1.028 
6 -0.273 
7 -0.197 
8 -0.668 
9 -0.793 
10 -0.428 
11 -0.967 
12 -0.557 
13 -0.205 
14 -0.401 
15 -0.436 
16 -0.250 


0.005 
0.006 


0.093 
0.073 
0.067 
0.000 


0.176 
0.199 
O95 
0.193 
0.215 
0.283 


0.282 
0.300 
0.298 
0.288 
0.325 
0.289 
0.274 
0.287 
0.289 
0.289 
0265 
0.338 
0.281 
0.278 
0.250 


0.000 
0.000 


0.000 
0.049 
0.000 
0.000 


0.077 
0.382 
0.510 
0.241 
0.331 
0.516 


0.211 
0587 
0.176 
0.000 
0.400 
0.495 
0.015 
0.006 
D132: 
0.001 
0.035 
0.545 
0.154 
0.116 
0.317 


-0.054 -0.036 
0.037 0.061 
-0.513 -0.147 
-0.286 0.000 
0.198 0.461 
0.000 0.000 
-0.655 0.033 
-0.564 0.216 
-0.254 0.511 
-0.152 0.606 
-0.631 0.213 
-0.737 0.370 
-0.906 0.200 
-0.750 0.424 
-0.987 0.181 
-1.591 -0.464 
-0.910 0.364 
-0.764 0.369 
-1.206 -0.131 
-1.356 -0.231 
-0.994 0.138 
-1.533 -0.402 
-1.077 -0.038 
-0.866 0.457 
-0.953 0.150 
-0.980 0.108 
-0.739 0.239 
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17 0.442 0.334 0.185 -0.212 1.096 
18 0.062 0.285 0.828 -0.497 0.621 
19) -0.044 0.283 0.876 -0.598 0.510 
20 -0.404 0.290 0.164 -0.972 0.165 
21 -0.682 0.296 0.021 -1.262 -0.101 
22 -0.645 0.266 0.015 -1.167 -0.123 
23 -0.155 0.284 0.585 -0.711 0.401 
24 -0.458 0.274 0.095 -0.996 0.080 
25 -0.644 0.287 0.025 -1.206 -0.082 
26 -1.205 0.313 0.000 -1.819 -0.591 
27 -0.685 0.297 0.021 -1.267 -0.103 
28 -0.461 0.255 0.071 -0.960 0.039 
29 -0.824 0.260 0.002 -1.334 -0.314 
30 -0.356 0.278 0.200 -0.902 0.189 
31 -0.647 0.240 0.007 “LAT? -0.177 
Intercept 2.663 0.287 0.000 2.100 3.227 
var(Int) 0.000 0.000 0.000 


Notes: Observations = 2,005, Course Sections = 80 


The next model adds course number by treatment interaction terms to the model of developmental 
education course grades for the combined cohorts. The largest estimated effect was for students in Math 


99, in which treatment students earned grades in developmental education courses 0.21 grade points 
higher than control group students in the same course number. This difference was 0.02 (0.189 points 
less) for students who enrolled in Engl 99 and 0.17 (0.040 points less) for students in Math 99. Neither 
of these estimates was statistically significant, and the post-estimation test found no evidence of 
significant variation in the treatment effect by course number (p = 0.469). 


Table 7: Treatment Model of Developmental Education Course Grades for Combined Cohorts, Treatment by Course 


Number Interaction 


8 SE p-value 95% Conf. Int. 

Treatment Group (Math 99) 0.206 O111 0.064 -0.012 0.425 
Course (Math 99) 

Engl 99 0.758 0.258 0.003 0.252 1.264 

Math 98 0.213 0.190 0.261 -0.158 0.585 
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Treatment by Course (Math 99) 
Engl 99 -0.189 0.162 0.242 = -0.507 0.128 
Math 98 -0.040 0152 0.791 -0.337 0.257 
Credits Attempted Pre-Intervention -0.045 0.005 0.000 -0.054 — -0.036 
Credits Earned Pre-Intervention 0.049 0.006 0.000 0.037 0.061 
Race/Ethnicity (White) 


Black 0.331 0.093 0,000 -0.514 ~—_-0.148 
Other 0.139 0.073 0.057 0.281 0.004 
Female (Male) 0.329 0.067. 0.000 0198 0.460 
Pell Received 0.000 0.000 0.000 0,000 ~—0.000 
Age Category (<18) 
18-21.99 0.302 0.176 0.086 0.646 0.043 
22-24.99 0.161 0199 0417 0.552 0.229 
25-29.99 0140 0.195 0.472 -0.242 0.523 
30-39.99 0.239 0.194 0.217 -0.141 0.618 
40-49.99 0.197 0.215 0361 0.619 0.225 
50+ 0.171 0.283 0.545. 0.725 (0.383 


Block ID (Block 1) 


2 -0.361 0.282 0.201 -0.914 0.193 
I) -0.158 0,299 0.597 -0.745 0.429 
4 -0.408 0.298 0.171 -0.992 0.176 
5 -1.038 0.288 0.000 -1.601  -0.474 
6 -0.272 0.325 0.402 -0.908 0.364 
7 0.267 0.233 0.252 -0.190 0.724 
8 -0.204 0.214 0.341 -0.624 0.216 
9 -0.329 0.230 0.152 -0.781 0.122 
10 0.032 0.233 0.889 -0.424 0.489 
11 -0.508 0.232 0.029 -0.963 — -0.053 
12 -0.090 0.203 0.658 -0.487 0.308 
13 0.255 0.291 0.381 -0.316 0.826 
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14 0.259 0.204 0.204 -0.141 0.659 

15 0.221 0.200 0.268 -0.171 0.614 
16 0.415 0.158 0.008 0106 0.724 
{7 1.114 0.276 0.000 = 0.573 1.655 
18 0.054 0.285 0.850 -0.505 0.613 
19 0.049 0.283 0.861 = 0.603 0.504 
20 0.410 0.290 0158 -0.979 0.159 
21 0.705 0.297 0.017 1.286 -0.124 
22 0.637 0.266 0.017 1459 -0.115 
23 0.307 0.226 »=0.174 = 0.136 ~—0.750 
24 0.004 0.215 0.985 -0.418 0.426 
25 0.182 0.230 0.428 0.633 0.268 
26 0.736 0.263 ~=—0,005— 1.252 -0.220 
27 0.218 0.244 0373 0.696 0.261 
28 0.000 (omitted) 
29 0.157 0176 0.371 0.501 0.187 
30 0.302 0.199 0129 -0.088 0.691 
31 0.000 (omitted) 

Intercept 1.962 0.216 0.000 1.539 2,384 

Var(Int) 0.000 0.000 0.000 


Notes: Observations = 2,005, Course Sections = 80 
Next-Semester Persistence 


The remaining analyses in the impact evaluation for this APR assess the next-semester outcomes for the 
combined cohorts. The sample for these analyses is delimited to students who were only enrolled in one 
of the three developmental education courses to prevent the sample contains students enrolled in both 
treatment and control group courses. The first analysis investigates the rates at which students persisted 
to the next long semester, defined as students in the Spring 2017 cohort enrolling in BPCC in Fall 2017 
and students in the Fall 2017 cohort enrolling in Spring 2018. This outcome is defined dichotomously 
(yes = 1, no =0), making the treatment effect estimates represent the percentage point increase in the 
likelihood of students persisting to the next semester. 


Table 8 contains the results of the first model which estimates an overall treatment effect across all three 
courses and does not contain treatment by course interactions. Students in the treatment group were 
estimated to be 2.5 percentage points more likely to persist to the next long semester, but this difference 
was not statistically significant (p = 0.247). 


66 | Page Giani Consulting & Evaluation, LLC 


Bossier Parish Community College First in the World Evaluation: Final Annual Performance Report 


Table 8: Treatment Model of Next Semester Persistence for Combined Cohorts, no Interaction 


p SE p-value 95% Conf. Int. 


Treatment Group 0.025 0.022 0.247 — -0.018 0.068 
Credits Attempted Pre-Intervention -0.013 0.002 0.000 = -0.015 -0.010 


Credits Earned Pre-Intervention 0.017 0.002 0.000 0.013 0.021 
Race/Ethnicity (White) 
Black -0.024 0.033 0.472 -0.089 0.041 
Other -0.010 0.025 0.679 -0.060 0.039 
Female (Male) 0.030 0.023 0192 -0.015 0.076 
Pell Received 0.000 0.000 0.000 0.000 0.000 
Age Category (<18) 
18-21.99 -0.050 0.064 0.437 -0.176 0.076 
22-24.99 -0.061 0.073 0.401 -0.203 0.081 
25-29.99 -0.014 0.071 0.838 -0.153 0.124 
30-39.99 0.034 0.070 0.630 -0.103 0.171 
40-49.99 0.003 0.077 0.974 -0.149 0.154 
50+ 0.025 0.095 0.795 -0.161 0.210 


Block ID (Block 1) 


2 0.061 0.107 0.571 -0.149 0.270 
3 0135 0.112 0.228 -0.084 0.354 
4 0.095 0.116 0.412 -0.132 0.322 
a 0.043 0.105 0.680 -0.162 0.249 
6 -0.099 0.176 0.573 -0.445 0.246 
7 0.137 0.103 0.185 -0.066 0.340 
8 0.079 0.099 0.425 -0.115 0.273 
9 0.199 0.103 0.052 -0.002 0.400 
10 0197 0.102 0.054 — -0.003 0.398 
11 0.165 0.101 0.103 -0.034 0.364 
12 0.180 0.095 0.058 -0.006 0.365 
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13 -0.121 0.186 0.516 -0.486 0.244 
14 -0.052 0.100 0.602 -0.247 0.143 
15 0.187 0.098 0.056 -0.005 0:379 
16 0.156 0.089 0.078 -0.018 0.330 
Le -0.058 0.155 0.707 -0.363 0.246 
18 0.273 0.104 0.009 0.069 0.478 
oe) 0.233 0.105 0.027 0.026 0.440 
20 0.156 0.101 0.120 -0.041 0.354 
21 0.125 0.108 0.247 -0.087 0.337 
22 0:125 0.095 0.187 -0.061 0.310 
23 0.221 0.098 0.024 0.029 0.414 
24 0.276 0.099 0.006 0.081 0.471 
25 0.203 0.103 0.048 0.002 0.405 
26 0.059 0.105 0573 -0.147 0.265 
27 0.107 0.100 0.288 -0.090 0.303 
28 0.138 0.090 0.125 -0.038 0.314 
22 0.188 0.090 0.037 0.011 0.365 
30 0.174 0.096 0.071 -0.015 0.362 
31 0.219 0.085 0.010 0.052 0.386 
Intercept 0.479 0.104 0.000 0.275 0.683 
var (Int) B.240* 1.89" 3.40°°° 3.08%° 


Notes: Observations = 1,714, Course Sections = 80 


The next model includes treatment by course interaction terms to explore whether the effect of access to 
the mobile apps on next-semester persistence varies across the three courses. The results show that 
treatment students in Math 99 were 9.0 percentage points more likely to persist to the next semester 
compared to control students, a statistically significant difference (p = 0.041). 


Treatment students in English 99 were 3.2 percentage points more likely to persist and treatment 
students were 2.8 percentage points less likely to persist compared to control group students. Although 
the point estimate suggests that the difference in the treatment effect varied significantly between Math 
98 and Math 99 students (p = 0.044), the post-estimation test found insufficient evidence to conclude 
there was significant variation in the treatment effect across courses (p = 0.152). 
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Table 9: Treatment Model of Next Semester Persistence for Combined Cohorts, Treatment by Course Number Interaction 


8 SE p-value 95% Conf. Int. 
Treatment Group (Math 99) 0.090 0.044 0.041 0.004 0.177 
Course (Math 99) 
Engl 99 0.167 0.094 0.078 -0.018 0.352 
Math 98 0.282 0.091 0.002 0105 0.460 
Treatment by Course (Math 99) 
Engl 99 0.058 0.056 0.303 0.169 0.052 
Math 98 0.116 0.057 0.044 0.228 ~—_-0.003 


Credits Attempted Pre-Intervention -0.012 0.002 0.000  -0.015 — -0.010 
Credits Earned Pre-Intervention 0.017 0.002 0.000 0.013 0.021 
Race/Ethnicity (White) 


Black 0.025 0.033 0.442 -0.090 0.039 
Other 0.013 0.025 0,606 -0.062 0.036 
Female (Male) 0.029 0.023 0.214 -0.017 0.074 
Pell Received 0.000 0.000 0.000 0,000 0.000 
Age Category (<18) 
18-21.99 0.053 0.064 0.409 0.179 0.073 
22-24.99 0.066 0.073 0.360 -0.208 0.076 
25-29.99 0.021 0.071 0.764 0.160 0.117 
30-39.99 0.027 0.070 0.696 -0.110 0.165 
40-49.99 0.005 0.077 0.953 0.156 0.147 
50+ 0.017 0.095 0.855 -0.168 0.203 


Block ID (Block 1) 


2 0.061 0.107 0.568 -0.148 0.271 
3 0.126 0.112 0.262 -0.094 0.345 
4 0.092 0.116 0.425 -0.135 0.319 
5 0.042 0.105 0.686 -0.163 0.248 
6 -0.090 0.176 0.608 -0.436 0.255 
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7 0.001 0.080 0,994 0.157 0.156 
8 0.059 0.074 0.428 0.203 0.086 
9 0.062 0.079 0.431 -0.092 0.216 
10 0.060 0.078 0.442 0.093 0.214 
ta 0.028 0.077 0.714 -0.123 0.179 
i? 0.043 0.068 0.524 ~=—--0.090 0.177 
13 0.258 0174 0137 0.599 0.083 
14 0.278 0.068 0.000 -0.412 -0.144 
15 0.036 0.066 0.583 0.166 0.094 
16 0.073 0.051 0.156 -0.173 0.028 
17 0.282 0.138 = 0.041. 0.551 ——-0.012 
18 0.272 0.104 0.009 0,068 0.476 
19 0.233 0.105 0.027 0.027 0.440 
20 0155 0.101 0123 -0.042 0.352 
21 0132 0.108 0.220 0.079 0.344 
22 0115 0.095 0.224 -0.070 0.301 
23 0.083 0.073 0.254 ~—--0.060 ~—0.226 
24 0.138 0.075 0.064 -0.008 0.285 
25 0.066 0.079 0.401 -0.089 0.221 
26 0.076 0.082 0.357 -0.237 (0.086 
27 0.029 0.076 0.698 -0.178 0.119 
28 0.000 (omitted) 
29 0.045 0.055 0.409 0.153 0.062 
30 0.051 0.063 0.418 0.175 0.073 
31 0.000 (omitted) 

Intercept 0.451 0.105 0.000 0.246 0.657 

Var(Int) 953° 4.36% 122° 743° 


Notes: Observations = 1,714, Course Sections = 80 
Next-Semester GPA 
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The next analyses investigate the effect of the treatment on the GPA students received in the next 
semester. The sample is further delimited to students who enrolled and attempted courses in the 
following semester (” = 1,176). GPA was calculated on a 4.0 scale as it was in previous analyses. The first 
model estimates the overall treatment effect across all three courses, the results of which ate found in 
Table 10. Students in the treatment group were estimated to earn a next-semester GPA 0.14 points 
higher than control group students, a marginally significant difference (p = 0.053). 


Table 10: Treatment Model of Next-Semester GPA for Combined Cohorts, no Interaction 


) SE p-value = 95% Conf. Int. 


Treatment Group 0.135 0.069 0.053 -0.002 0.271 
Credits Attempted Pre-Intervention —_-0.033 0.005 0.000 -0.044 -0.023 
Credits Earned Pre-Intervention 0.044 0.007 0.000 0.030 0.057 
Race/Ethnicity (White) 


Black -0.303 0.106 0.004 -0.510 -0.096 
Other -0.096 0.079 0.220 -0.251 0.058 
Female (Male) 0.105 0.074 0.155 -0.040 0.250 
Pell Received 0.000 0.000 0.000 0.000 0.000 
Age Category (<18) 
18-21.99 0.001 0.199 0.994 -0.388 0.390 
22-24.99 0.229 0.227 0.312 -0.216 0.674 
25-29.99 0.096 0.218 0.661 -0.332 0.523 
30-39.99 0.461 0.216 0.033 0.038 0.883 
40-49.99 0.424 0.238 0.075 -0.042 0.891 
50+ 0.268 0.283 0.344 -0.286 0.822 


Block ID (Block 1) 


2 -0.572 0.372 0.124 -1.300 0.157 
3 -0.271 0.380 0.475 -1.017 0.474 
4 -0.504 0.395 0.202 -1.278 0.270 
5 -0.477 0.377 0.206 -1.216 0.262 
6 -0.985 0.725 0.174 -2.407 0.437 
7 -0.126 0.351 0.720 -0.814 0.563 
8 0.201 0.348 0.564 -0.481 0.883 
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9 0.291 0.344 0.399 -0.385 0,966 
10 0.273 0.343 0.426 -0.400 0.946 
11 0.557 0.341 0102 -1.226 0.111 
12 0.181 0.324 0.576 -0.454 0,816 
13 0.450 0,862 0.602 -2140 1.240 
14 0.539 0.359 0133 -0.164 1.242 
15 0.457 0.334 O171 0.197 1411 
16 0.061 0.308 0.844  -0.543 0,664 
17 -0.900 0.587. 0.125. 2.051 0.250 
18 0.591 0.345. 0.087. 1.267 0.085 
19 0.626 0.351 0.074 1.314 0.062 
20 0.654 0.343. «0.057 1.326 ~——0.019 
21 -0.283 0.370 = 0.443. 1.008 ~— 0.441 
22 0.349 0.325. (0,284.— -0.986 —0.289 
23 0.107 0.331 0.746 -0.542 0,757 
24 0.061 0.330 0.854 -0.585 0.707 
25 0.186 0.343 0.587  -0.486 0.859 
26 0.122 0.367 —s«0.738~— 0.841 0.596 
27 0.030 0.344 0.931  -0.645 0.705 
28 0.193 0.312 (0.537 -0.805. 0.419 
29 0.075 0313 0811  -0.539 0.688 
30 0.167 0.328 0.610 -0.475 — 0,809 
31 0.172 0.297 0.563 -0.411 0.754 
Intercept 1.952 0.350 0.000 1.266 —-2.637 
var(Int) 6.85°25 — 4,07624 5.972% 7,86e20 


Notes: Observations = 1,176, Course Sections = 80 


The next model, found in Table 11, adds treatment by course number interaction terms to the model of 
next-semester GPA to examine whether the effect varies significantly across the three courses. The 
results suggest that the effect of the mobile apps on next-semester GPA was concentrated among 
students in Math 99, in which treatment students recetved a GPA 0.37 points higher than students in the 
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control group, a statistically significant difference (p = 0.001). The effect was minimal for students in the 
other two courses. 


Treatment students in English 99 earned a next-semester GPA 0.03 points higher than control students 
and treatment students in Math 99 earned a next-semester GPA 0.02 points less than control students. 
The point estimates suggest that the effect of the treatment did vary significantly across courses, and the 
post-estimation test similarly found significant variation in the treatment effect by course number (p = 
0.032). 


Table 11: Treatment Model of Next Semester GPA for Combined Cohorts, Treatment by Course Number Interaction 


8 SE p-value 95% Conf. Int. 
Treatment Group (Math 99) 0.370 0.114 0.001 0147 0.593 
Course (Math 99) 
Engl 99 0.047 0.319 0.883. -0.579 0.673 
Math 98 0.115 0195 0.557 0.497 0.268 
Treatment by Course (Math 99) 
Engl 99 0.343 0.185 0.063 -0.704 0.019 
Math 98 0.394 0.158 0.013 -0.704 —-0.084 


Credits Attempted Pre-Intervention -0.033 0.005 0.000 = -0.043 —-0.023 


Credits Earned Pre-Intervention 0.043 0.007 0.000 0.029 0.057 
Race/Ethnicity (White) 
Black -0.289 0.106 0.006 -0.496 = -0.082 
Other -0.090 0.078 0.253 -0.244 0.064 
Female (Male) 0.112 0.074 0.127 -0.032 0.257 
Pell Received 0.000 0.000 0.000 0.000 0.000 
Age Category (<18) 
18-21.99 0.007 0.198 0.973 -0.382 0.395 
22-24.99 0.241 0.227 0.288 -0.204 0.686 
25-29.99 0.114 0.218 0.603 -0.314 0.541 
30-39.99 0.490 0.215 0.023 0.067 0.912 
40-49.99 0.446 0.238 0.061 -0.020 0.913 
50+ 0.292 0.282 0.301 -0.261 0.845 


Block ID (Block 1) 
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y, 0.572 0.371 0.123 1.298 0.155 
3 0.267 0.379 0,482 -1.010 0.476 
4 0.504 0.394. 0.200 1.276 0.268 
5 0.478 0.376 0.204 1.215 0.259 
6 1.024 0.725 0158 -2.444 0.396 
7 0.044 0.252 0.862 -0.450 0.537 
8 0.383 0.247 0.121 += -0.101 0.866 
9 0.474 0.241 0.050 0.001 ~—0.946 
10 0.453 0.241 0.060 -0.020 0.926 
ta 0.372 0.239. 0.120» 0.841 0.097 
{2 0.328 0.213 0.123 -0.089 0.746 
13 0.268 0.825 0.746 -1.886 1.350 
14 0.433 0.242 + ~=0.073-~—-0.041 ~—-0.908 
15 0.333 0.204 0.102 -0.066 0.731 
16 0.057 0157 0.715 0.365 0.250 
i7 1.008 0524 0,054 -2.034 0.019 
18 0.596 0.344 0,083 1.270 ~—-0.079 
19 0.624 0.350 0.075 1.310 0.062 
20 0.657 0.343 0.055 -1.329 0.014 
21 0.317 0371 0393 -1.045 0.411 
22 0.351 0.324 0.279 0.986 0.284 
23 0.291 0.223 0193 -0.147 0.729 
24 0.229 0.222 0.303 ~=—--0.207_—0.664 
25 0.354 0.241 0.142 -0.118 0.826 
26 0.015 0.274 0.957 -0.523 0.552 
oF 0.192 0.244 0432 -0.287 0.671 
28 0.000 (omitted) 

29 0.018 0169 0.916 0.348 0.313 
30 0.041 0.191 0.832 -0.335 0.416 
31 0.000 (omitted) 
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Intercept 1.57. 0.232 0.000 1,502 2.412 


Var(Int) LOB? Aare oars - eles 


Notes: Observations = 1,176, Course Sections = 80 
Next-Semester Course Pass Rates 


The next analysis assesses the effect of the treatment on students’ next-semester course pass rates, 
defined as the percentage of attempted courses that students passed (0-100%). The sample is once again 
delimited to students who enrolled in the subsequent semester and attempted courses. The results in 
Table 12 show the overall treatment effect across all three course numbers. Students in the treatment 
group were estimated to pass 3.7 percentage points more of their courses compared to students in the 
control group, a marginally significant difference (p = 0.074). 


Table 12: Treatment Model of Next Semester Course Pass Rates for Combined Cohorts, no Interaction 


) SE p-value 95% Conf. Int. 


Treatment Group 0.037 0.021 0.074 -0.004 0.078 
Credits Attempted Pre-Intervention —-0.010 0.002 0.000 -0.013 -0.007 


Credits Earned Pre-Intervention 0.014 0.002 0.000 0.010 0.018 
Race/Ethnicity (White) 
Black -0.062 0.032 0.051 -0.124 0.000 
Other -0.014 0.024 0.547 -0.060 0.032 
Female (Male) 0.025 0.022 0.265 -0.019 0.068 
Pell Received 0.000 0.000 0.003 0.000 0.000 
Age Category (<18) 
18-21.99 -0.057 0.060 0.341 -0.174 0.060 
22-24.99 -0.021 0.068 0.759 -0.154 0.113 
25-29.99 -0.037 0.065 0.568 -0.166 0.091 
30-39.99 0.017 0.065 0.796 -0.110 0.143 
40-49.99 0.009 0.071 0.895 -0.131 0.149 
50+ 0.004 0.085 0.962 -0.162 0.170 


Block ID (Block 1) 


2 -0.183 OT 0.101 -0.402 0.036 
a) -0.047 0.114 0.683 -0.270 0.177 
4 -0.038 0.118 0.747 -0.271 0.194 
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5 0.121 0113 0.285 -0.343 0.101 
6 0.263 0.218 0.227 -0.690 0.163 
7 -0.026 0.105 0.806 -0.233 0,181 
8 0.094 0.104 0.367 -0.111 0,299 
9 0.105 0.103 0.310 -0.098 0.308 
10 0.066 0.103 0.520 -0.136 0.268 
11 0.108 0.102 0.291 -0.309 0,092 
12 0.086 0.097 0.374 -0.104 0.277 
13 -0.033 0.259 0,900 -0.540 0.475 
14 0.174 0.108 0.107 -0.037 0.384 
15 0.110 0.100 0.274 -0.087 0.306 
16 0.041 0.092 0.655 0.140 0,222 
17 0.146 0176 0.408 -0.491 0.200 
18 0.140 0.103 0176 -0.343 0.063 
19 0.150 0.105 0.156 -0.356 0.057 
20 -0.226 0.103 0.028 -0.428  -0.024 
21 0.085 0111 0.442 -0303 0,132 
22 0.061 0,098 0.532 -0.252 0,130 
23 0.038 0.099 0.702 -0.157 0.233 
24 0.028 0.099 0.779 0.166 0.222 
25 0.095 0.103 0.354 -0.106 0.297 
26 -0.033 0.110 0.766 -0.248 0.183 
2 0.013 0.103 0.902 -0.190 0.215 
28 -0.033 0.094 0.722 -0.217 0,150 
29 0.051 0.094 0.583 0.133 0.236 
30 0.043 0.098 0.662 -0.150 0.236 
31 0.062 0.089 0.489 -0.113 0.236 

Intercept 0.707 0.105 0.000 0.501 0.913 

var(Int) 9356. 401 0 


Notes: Observations = 1,176, Course Sections = 80 


76 | Page Giani Consulting & Evaluation, LLC 


Bossier Parish Community College First in the World Evaluation: Final Annual Performance Report 


The next analysis adds course number by treatment interaction terms to the model of next-semester 
course pass tates to examine whether this effect varies significantly across groups. The largest estimated 
effect was for students in Math 99, in which treatment students passed 8.2 percentage points more of 
their courses in the next semester compared to control students, a statistically significant difference (p = 


0.016). 


Treatment students in English 99 passed 0.1 percentage points fewer courses than control students and 
treatment students in Math 98 passed 1.9 percentage points more courses than control students. Neither 
of these estimates was statistically significant, and the post-estimation test found no evidence of 
significant variation in the treatment effect by course number (p = 0.225). 


Table 13: Treatment Model of Next Semester Course Pass Rates for Combined Cohorts, Treatment by Course Number 
Interaction 


6B SE p-value 95% Conf. Int. 
Treatment Group 0.082 0.034 0.016 0.015 0.149 
Course (Math 99) 
Engl 99 -0.005 0.096 0955 --0:T9S- «0183 
Math 98 -0.055 0.059 0.349  -0.170 0.060 
Treatment by Course 
Engl 99 -0.088 0.055 0.113 = -0.197 0.021 
Math 98 -0.063 0.048 0.184 -0.156 0.030 


Credits Attempted Pre-Intervention -0.010 0.002 0.000 -0.013 -0.007 
Credits Earned Pre-Intervention 0.014 0.002 0.000 0.009 0.018 
Race/Ethnicity (White) 


Black 0.060 0.032 0,060 0.122 0.003 
Other 0.012 0.024 0.604 -0.058 0.034 
Female (Male) 0.026 0.022 0.244 -0.018 0.069 
Pell Received 0.000 0.000 0.002 0,000 0.000 
Age Category (<18) 
18-21.99 0.054 0.060 0369 0.170 0.063 
22-24.99 0.016 0.068 0815 -0.150 0.118 
25-29.99 0.032 0.066 0.629 0.160 0.097 
30-39.99 0.024 0.065 0.710 -0.103 0.151 
40-49.99 0.016 0.072 0.822 -0.124 0.156 
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50+ 0.010 0.085 0.909  -0.157 0.176 
Block ID (Block 1) 
2 -0.185 0.111 0.098 -0.403 — 0.034 


3 -0.046 0.114 0.687 -0.269 0.177 
4 -0.038 0.118 0.747 -0.270 0.194 
5 -0.122 0.113 O27) -0.344 0.099 
6 -0.278 0.218 0.202 -0.705 0.149 
7 0.005 0.076 0.948 -0.143 0.153 
8 0.126 0.074 0.089 -0.019 0.272 
> 0.138 0.072 0.058 -0.005 0.280 


10 0.098 0.073. 0178 -0.044 0.240 
11 0.076 0.072 0.291 0.217 0.065 
12 0115 0.064 0.073 -0.011 0.240 
13 0.000 0.248 1.000 -0.486 0.486 
14 0.125 0.073 0.086 -0.018 0.267 
15 0.057 0.061. 0.351. -0.063 0.177 
16 0.010 0.047 0.831 0.102 0.082 
17 0.195 0157 0.216 0.503 0.114 
18 0.144 0103 0165 0.346 0.059 
19 0.151 0105 0152 0.357 0.056 
20 0.230 0103 0.026 -0.431  -0.028 
21 0.100 0112 0371 0.319 0.119 
9 0.062 0.097 0523 0.253 0.129 
23 0.071 0.067 0.292 -0.061 ~—0.202 
24 0.058 0.067. —0.381.~—-0.072 0.189 
25 0.126 0.072 0.082 -0.016 0.268 
26 0.006 0.082 0,944 0.167 0.156 
a 0.043 0.073. 0.561 -0.101_~—0.187 
28 0.000 (omitted) 

29 0.005 0.051 0.924 0.094 0.104 
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30 0.010 0.058 0.865 0.123 0.103 
31 0.000 (omitted) 

Intercept 0.735 0.070 0.000 0.598 0.872 

Var(Int) 26 oi 2.93°% — 2.33°! 


Notes: Observations = 1,176, Course Sections = 80 
Next-Semester Credits Earned 


The final analysis estimates the effect of access to the mobile apps on the number of credits students 
earned in the subsequent semester. In this instance, the sample includes all students who were enrolled in 
only one intervention course (# = 1,714), and students who did not persist to the next semester ate 
treated as having earned zero credits. The results of the model found in Table 14 show that, across the 
three developmental education courses, students given access to the mobile apps earned close to half a 
credit (0.46) more than students in the control group, a marginally significant effect (p = 0.072). 


Table 14: Treatment Model of Next Semester Credits Earned for Combined Cohorts, no Interaction 


B SE p-value = 95% Conf. Int. 


Treatment Group 0.458 0.254 0.072 — -0.040 0.956 
Credits Attempted Pre-Intervention -0.167 0.018 0.000 -0.202  -0.132 


Credits Earned Pre-Intervention 0.231 0.024 0.000 0.184 0.278 
Race/Ethnicity (White) 
Black -0.793 0.385 0.039 -1.547 -0.039 
Other -0.300 0.292 0.303 -0.872 0.271 
Female (Male) 0.335 0.270 0.190 -0.175 0.885 
Pell Received 0.000 0.000 0.055 0.000 0.000 
Age Category (<18) 
18-21.99 -0.936 0.748 0.210 -2.402 0.529 
22-24.99 -1.194 0.843 O:157 -2.847 0.458 
25-29.99 -0.978 0.821 0.234 -2.587 0.632 
30-39.99 -0.560 0.814 0.491 -2.155 1.034 
40-49.99 -0.602 0.899 0.503 -2.363 1.160 
50+ -0.918 1,099 0.403 -3.073 1.236 


Block ID (Block 1) 
Z -0.253 1.244 0.839 = -2.692 2.186 
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0.567 1.300 0.663 -1.982 3.115 
0.241 1.347 0.858 -2.400 2.882 
-0.220 L219 0.857 -2.609 2.169 


0.970 1.202 0.420 -1.386 225 


3 
4 
5 
6 -1.403 2.050 0.494 —-5.420 2.614 
7 
8 1,232 1154 0.285  —-1.024 3.487 
9 


2.700 1.192 0.023 0.364 5.036 


10 2.506 1.190 0.035 0.174 4.837 
11 0.197 1.178 0.867 -2.112 2.506 
12 1.893 1.101 0.086 -0.265 4.051 
13 0.166 2.165 0.939 -4.078 4.410 
14 0.962 Tol Sz 0.406 -1.305 3.229 
15 2.566 1.138 0.024 0.336 4.796 
16 1502 1.030 0.145 -0.516 Doel 
sia -0.899 1.805 0.619 -4.437 2.640 
18 0.744 1,212 0.539 -1.631 3.119 
phe) Tel 1.226 0.365 ef, 292 3.514 
20 -0.761 1.170 0.516 -3.054 L539 
21 0.693 1255 0.581 -1.767 D155 
22 0.912 1.100 0.407 -1.244 3.069 
23 20315 1.142 0.043 0.075 4.550 
24 2.416 1.156 0.037 0.150 4.681 
25 2173 1.196 0.069 -0.171 4.517 
26 0.386 Lae O52 -2.009 2.780 
27 0.365 1.165 0.754 = -1.918 2.647 
28 0,995 1.043 0.340 -1.050 3.041 
29 2.546 1.049 0.015 0.490 4.601 
30 2.011 olla 0.072 -0.179 4.201 
31 2.635 0.992 0.008 0.692 4.579 
Intercept 4.324 1.207 0.000 1.958 6.690 
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var (Int) 108°) “93a? 1.87°'7 — 2.08°° 


Notes: Observations = 1,714, Course Sections = 80 


The model found in Table 15 below adds course number by treatment interaction terms to the model of 
next semester credits earned. The largest estimated effect was for students in Math 99, in which 
treatment students earned 0.55 more credits in the following semester compared to control students. 
However, this difference was not statistically significant, nor were the interaction effects exploring 
whether the treatment effect varied across the three courses. The post-estimation test found no evidence 
of significant variation in the treatment effect by course number (p = 0.925). 


Table 15: Treatment Model of Next Semester Credits Earned for Combined Cohorts, 
Treatment by Course Number Interaction 


8 SE p-value 95% Conf. Int. 
Treatment Group 0.547 0.422 0.195 -0.280 1.374 
Course (Math 99) 
Engl 99 -2.496 1.056 0.018  -4.565 = -0.428 
Math 98 -1.599 0.713 0.025 -2.996  -0.202 
Treatment by Course 
Engl 99 -0.259 0.667 0.698 — -1.567 1.049 
Math 98 -0.067 0.588 0.910 — -1.219 1.086 


Credits Attempted Pre-Intervention -0.167 0.018 0.000  -0.202 = -0.132 
Credits Earned Pre-Intervention 0.231 0.024 0.000 0.184 0.278 
Race/Ethnicity (White) 


Black 0.791 0.385 0.040 1.546 -0.036 
Other 0.294 0.292 0.313 0.867 0.278 
Female (Male) 0.358 0.271 0.186 -0.173 0.888 
Pell Received 0.000 0.000 0.058 0,000 0.000 
Age Category (<18) 
18-21.99 0.927 0.748 0.215 -2.394 0.540 
22-24.99 1.181 0.844 0162 -2.835 0.474 
25-29.99 0.961 0.822 0.242 = -2.572 0.650 
30-39.99 0.547 0.814 0.502 -2.143 1.049 
40-49.99 0.585 0.900 0516 -2.348 1.179 
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50+ -0.905 1.100 0.411 -3.062 15h 
Block ID (Block 1) 
2 -0.255 1.244 0.838 -2.693 2.184 


3 0591 1302 0.650 1.961 3.143 
4 0.249 1348 0.853 -2.392 2.890 
5 0.218 1.219 0.858 -2.607 2.170 
6 1.426 2.050 0487 5.445 2.593 
7 0.023 0.929 0,981 1.842 1.797 
8 0.239 0.860 0.781 -1.447 1.925 
9 1.706 0.914 0.062 -0.085 3.497 
10 1.509 0.913 0.098 -0.280 3,298 
11 0.801 0.897 0372 -2.559 0.957 
12 0.902 0.794 0.256 -0.654 2.458 
13 0.825 2.024 0.684 -4.793 3.143 
14 1.661 0.795 0.037 -3.220~—-0.102 
15 0.061 0.772 0.937 1.573 1.451 
16 1.116 0595 0,061 -2.282 0.051 
17 3.527 1.602 0.028  -6.668  -0.387 
18 0.748 1.212 0537 -1.627 3.123 
19 1.110 1.226 0.365 -1.293 3.512 
20 0.758 1170 0517  -3.051 1.536 
21 0.673 1.256 0.592 -1.789 3.135 
9 0.938 1102 0.395 -1.222 3.098 
23 1.319 0.848 0.120 -0.344 2.982 
24 1.422 0.871 0.103 -0.285 3,129 
25 1.179 0,919 0.200 -0.622 2.980 
26 0.604 0.958 0528 -2.481 1.273 
a 0.627 0.884 0478 -2.360 1.106 
28 0.000 (omitted) 

29 0.066 0.638 0.918 1.317 1.185 
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30 0.614 0.736 0.404 -2.056 0.827 
31 0.000 (omitted) 

Intercept 6.893 0.869 0.000 5190 8.596 

Var(Int) 3.642 5.36" 0 


Notes: Observations = 1,714, Course Sections = 80 
Credential Attainment Rates 


The final outcome examined in this evaluation of the effect of the mobile apps on student outcomes is 
credential attainment. We investigate three separate outcomes related to credential attainment: the receipt 
of any credential (associate’s or certificate), associate’s receipt, and certificate receipt. 


Table 16 presents the results of the model estimating the effect of access to the mobile apps on the 
receipt of any credential. No treatment by course interaction term is present in the model. Students in the 
treatment group were estimated to be 0.8% mote likely to complete any credential compared to students 
in the control group, but this difference was not statistically significant (p = 0.618). 


Table 16: Treatment Model of Any Credential Attainment for Combined Cohorts, no Interaction 


B SE p-value 95% Conf. Int. 

Treatment Group 0.008 0.016 0.618 -0.024 0.040 
Credits Attempted Pre-Intervention -0.008 0.001 0.000 -0.011 -0.006 
Credits Earned Pre-Intervention 0.017 0.002 0.000 0.014 0.020 
Race/Ethnicity (White) 

Black -0.030 0.025 0.221 -0.079 0.018 

Other -0.043 0.019 0.024 -0.080 -0.006 
Female (Male) -0.004 0.017 0.830 -0.038 0.031 
Pell Received 0.000 0.000 0.710 0.000 0.000 
Age Category (<18) 

18-21.99 0.026 0.048 0.591 -0.069 0.121 

22-24.99 -0.018 0.055 0.746 -0.125 0.089 

25-29.99 0.009 0.053 0.866 -0.095 0.113 

30-39.99 0.011 0.053 0.828 -0.092 0.115 

40-49.99 0.068 0.058 0.242 -0.046 0.182 

50+ -0.096 0.071 0.179 -0.235 0.044 
Block ID (Block 1) 

2 -0.092 0.080 0.253 -0.250 0.066 
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3 -0.066 0.084 0.429 -0.231 0.098 
4 -0.146 0.087 0.093 -0.317 0.025 
5 -0.053 0.079 0.504 -0.207 0.102 
6 -0.180 0.133 0175 -0.439 0.080 
7 -0.019 0.078 0.812 -0.171 0.134 
8 -0.055 0.074 0.456 -0.201 0.090 
. 0.015 0.077 0.848 -0.136 0.166 
10 -0.044 0.077 0.567 -0.195 0.107 
AT -0.161 0.076 0.034 -0.311 -0.012 
12 -0.080 0.071 0.262 -0.219 0.060 
13 -0.021 0.140 0.881 -0.295 0.253 
14 -0.051 0.075 0.494 -0.198 0.095 
15 -0.049 0.074 0.506 -0.193 0.095 
16 -0.027 0.067 0.681 -0.158 0.103 
17 -0.240 0.117 0.040 -0.469 -0.011 
18 -0.136 0.078 0.084 -0.289 0.018 
19 -0.130 0.079 0.102 -0.285 0.026 
20 -0.115 0.076 0.129 -0.263 0.033 
21 -0.110 0.081 OLS -0.270 0.049 
22 -0.102 0.071 0.153 -0.241 0.038 
23 -0.098 0.074 0.185 -0.243 0.047 
24 -0.029 0.075 0.696 -0.176 0.117 
25 -0.088 0.077 0.254 -0.240 0.063 
26 -0.112 0.079 0.158 -0.266 0.043 
27 -0.093 0.075 0.218 -0.240 0.055 
28 -0.124 0.067 0.065 -0.257 0.008 
29 -0.076 0.068 0.263 -0.209 0.057 
30 -0.123 0.072 0.088 -0.265 0.018 
31 -0.042 0.064 0.516 -0.167 0.084 
Intercept 0.160 0.078 0.041 0.007 0.313 
var(Int) 3.00" 1.42° 274 20s 
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Notes: Observations = 1,714, Course Sections = 80 


The next model adds treatment by course number interaction terms to explore if the effect of the 
treatment on credential attainment varied across the three courses. No significant differences were found 
between courses in the treatment effect. This finding was corroborated by the post-estimation test (p = 


0.631). 
Table 17: Treatment Model of Any Credential Attainment for Combined Cohorts, 


Treatment by Course Number Interaction 


6 SE p-value 95% Conf. Int. 
Treatment Group 0.017 0.033 0.611 -0.048 0.082 
Course (Math 99) 
Engl 99 -0.111 0.071 0.118 -0.250 0.028 
Math 98 -0.044 0.068 0.517 -0.178 0.090 
Treatment by Course 
Engl 99 -0.025 0.042 0.556 -0.108 0.058 
Math 98 0.003 0.043 0.949 -0.082 0.087 
Credits Attempted Pre-Intervention -0.008 0.001 0.000 -0.011 -0.006 
Credits Earned Pre-Intervention 0.017 0.002 0.000 0.014 0.020 
Race/Ethnicity (White) 
Black -0.043 0619 0.024 -0.080 — -0.006 
Other -0.030 0.025 0.233 -0.079 0.019 
Female (Male) -0.004 0.017 0.836 -0.038 0.031 
Pell Received 0.000 0.000 0.732 0.000 0.000 
Age Category (<18) 
18-21.99 0.025 0.048 0.606 -0.070 0.120 
22-24.99 -0.018 0.055 0.739 -0.125 0.089 
25-29.99 0.009 0.053 0.872 -0.096 0.113 
30-39.99 0.012 0.053 0.819 -0.091 0.115 
40-49.99 0.068 0.058 0.245 -0.046 0.182 
50+ -0.094 0.071 0.186 -0.233 0.045 
Block ID (Block 1) 
2 -0.092 0.080 0.255 -0.249 0.066 
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3 -0.068 0.084 0.422 -0.233 0.097 
4 -0.147 0.087 0.092 -0.318 0.024 
5 -0.053 0.079 0.504 -0.207 0.102 
6 -0.179 0.133 0.177 -0.439 0.081 
7 0.105 0.060 0.081 -0.013 0222 
8 0.068 0.056 0.222 -0.041 0.177 
. 0.138 0.059 0.019 0.022 0.254 


10 0.080 0.059 0173 -0.035 0.196 
11 0.037 0.058 0.526 0.150 0.077 
12 0.041 0.051 0.420 -0.059 0.142 
13 0101 0.131 0438 -0.155 0.358 
14 0.008 0.051 0,872 0.109 0.092 
15 0.007 0.050 0,896 0.104 0.091 
16 0.016 0.038 0.675 -0.059 0,092 
17 0.198 0.104 0.057 -0.401 0.005 
18 0.136 0.078 0.084  -0.289 0.018 
19 0.129 0.079 0.103 0.285 0.026 
20 0.115 0.076 0129 0.263 0.034 
21 0.109 0.081 0179 0.268 0.050 
oD 0.103 0.071 0.149 0.243 0.037 
23 0.026 0.055 0.635 -0.081 0.133 
24 0.094 0.056 0.095 -0.016 0.204 
25 0.035 0.059 0.554 -0.081 0.152 
26 0.008 0.062 0.900 -0.114 0.129 
Os 0.029 0.057 0.618 -0.084 0.141 
28 0.000 (omitted) 
29 0.031 0.041 0.452 0.112 0.050 
30 0.081 0.048 0.090 -0.174 0.013 
31 0.000 (omitted) 

Intercept 0.156 0.079 0.049 0.001 0.311 

Var(Int) 0.000 0.000 0.000 0,000 
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Notes: Observations = 1,714, Course Sections = 80 


We next examined if being given access to the mobile apps affected students’ likelihood of associate’s 
degree attainment. These results are found in Table 18. Students in the treatment group were 0.8% more 
likely to complete an associate’s degree during the study timeframe compared to the control group, a 
non-significant difference (p = 0.506). 


Table 18: Treatment Model of Associate’s Degree Attainment for Combined Cohorts, no Interaction 


6 SE p-value 95% Conf. Int. 
Treatment Group 0.008 0.013 0.506 -0.016 0.033 
Credits Attempted Pre-Intervention -0.008 0.001 0.000 -0.010 -0.006 
Credits Earned Pre-Intervention 0.013 0.001 0.000 0.011 0.016 
Race/Ethnicity (White) 
Black -0.039 0.014 0.007 -0.067 -0.011 
Other -0.002 0.019 0.917 -0.039 0.035 
Female (Male) -0.005 0.013 0.704 -0.031 0.021 
Pell Received 0.000 0.000 0.648 0.000 0.000 
Age Category (<18) 
18-21.99 0.038 0.037 0.309 -0.035 0.110 
22-24.99 0.028 0.042 0.502 -0.054 0.110 
2522999 0.042 0.041 0.297 -0.037 0.122 
30-39.99 0.058 0.040 0.147 -0.021 0.137 
40-49.99 0.049 0.044 0.271 -0.038 0.136 
50+ -0.001 0.054 0.986 -0.107 0.106 
Block ID (Block 1) 
2 -0.041 0.062 0.507 -0.161 0.080 
3 -0.049 0.064 0.448 -0.175 0.077 
4 -0.024 0.067 0.721 -0.154 0.107 
5 -0.030 0.060 0.624 -0.148 0.089 
6 -0.061 0.101 0.545 -0.260 0.137 
7 0.032 0.059 0.595 -0.085 0.148 
8 0.052 0.057 0.363 -0.060 0.163 
2 -0.001 0.059 0.984 -0.117 0.114 
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10 0.003 0.059 0.954 -0.112 0.119 
11 -0.033 0.058 0.568 -0.147 0.081 
il -0.010 0.054 0.855 -0.117 0.097 
1 -0.030 0.107 0.780 -0.240 0.180 
14 0.050 0.057 0.385 -0.062 0.162 
15 0.040 0.056 0.481 -0.071 0.150 
16 0.058 0.051 0.253 -0.042 0.158 
Av -0.089 0.089 0.318 -0.264 0.086 
18 -0.043 0.060 0.474 -0.160 0.075 
19 -0.034 0.061 O:5 71 -0.153 0.084 
20 -0.037 0.058 521 -0.151 0.076 
21 -0.011 0.062 0.860 -0.133 0.111 
22 -0.027 0.054 0.614 -0.134 0.079 
23 -0.038 0.056 0.496 -0.149 0.072 
24 -0.042 0.057 0.458 -0.154 0.070 
25 -0.027 0.059 0.643 -0.143 0.088 
26 -0.012 0.060 0.847 -0.130 0.107 
2 -0.039 0.058 0.493 -0.152 0.073 
28 -0.040 0.052 0.443 -0.141 0.062 
29 0.007 0:052 0.896 -0.095 0.108 
30 -0.028 0.055 0.613 -0.136 0.080 
31 0.037 0.049 0.451 -0.059 0.133 
Intercept 0.032 0.060 0.590 -0.085 0.149 
var (Int) 2.44E-26 1.11E-25 3.22E-30 1.84E-22 


Notes: Observations = 1,714, Course Sections = 80 


The next model explore whether the effect of the mobile apps on associate’s degree attainment varied 
across the three courses by including treatment by course number interaction terms. The results of this 
model are found in Table 19. The largest effect on associate’s degree attainment was found for students 
in Math 99, who were 1.6% mote likely to earn an associate’s degree if they were in the treatment group. 
However, this difference was not statistically significant, nor were the interaction terms exploring if this 
effect varied across courses. The post-estimation test also found no significant treatment by course 
number interaction (p = 0.484). 
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Table 19: Treatment Model of Associate’s Degree Attainment for Combined Cohorts, 


Treatment by Course Number Interaction 


B SE p-value 95% Conf. Int. 

Treatment Group 0.016 0.021 0.438 -0.025 0.057 
Course (Math 99) 

Engl 99 -0.025 0.052 0.634 -0.127 0.077 

Math 98 -0.073 0.035 0.039 -0.142 -0.004 
Treatment by Course 

Engl 99 -0.023 0.033 0.493 -0.087 0.042 

Math 98 -0.006 0.029 0.838 -0.063 0.051 
Credits Attempted Pre-Intervention -0.008 0.001 0.000 -0.010 -0.006 
Credits Earned Pre-Intervention 0.013 0.001 0.000 0.011 0.016 
Race/Ethnicity (White) 

Black -0.039 0.014 0.008 -0.067 -0.010 

Other -0.002 0.019 0.924 -0.039 0.036 
Female (Male) -0.005 0.013 0.718 -0.031 0.021 
Pell Received 0.000 0.000 0.625 0.000 0.000 
Age Category (<18) 

18-21.99 0.038 0.037 0.299 -0.034 0.111 

22-24.99 0.029 0.042 0.484 -0.053 0.111 

25-29.99 0.044 0.041 0.281 -0.036 0.124 

30-39.99 0.059 0.040 0.140 -0.019 0.138 

40-49.99 0.050 0.044 0.257 -0.037 0.138 

50+ 0.000 0.054 0.997 -0.106 0.107 
Block ID (Block 1) 

2 -0.041 0.062 0.505 -0.162 0.080 

) -0.047 0.064 0.468 -0.173 0.080 

4 -0.023 0.067 0.729 -0.154 0.108 

5 -0.029 0.060 0.625 -0.148 0.089 

6 -0.063 0.101 0.532 -0.262 0.135 

7 0.071 0.046 0.120 -0.019 0.161 
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8 0.092 0.043 0.031 0.008 0.175 

9 0.039 0.045 0.394 -0.050 0.127 
10 0.043 0.045 0.343 -0.046 0.131 
11 0.006 0.044 0.891 -0.081 0.093 
12 0.030 0.039 0.445 -0.047 0.107 
13 0.010 0.100 0.921 -0.186 0.206 
14 0.014 0.039 0.726 -0.063 0.091 
15 0.003 0.038 0.928 -0.071 0.078 
16 0.023 0.029 0.440 -0.035 0.080 
17 -0.125 0.079 0.113 -0.281 0.030 
18 -0.043 0.060 0.478 -0.160 0.075 
19 -0.034 0.061 0.570 -0.153 0.084 
20 -0.037 0.058 0.524 -0.150 0.076 
21 -0.013 0.062 0,838 -0.134 0.109 
22 -0.025 0.055 0.644 -0.132 0.082 
23 0.001 0.042 0.975 -0.081 0.084 
24 -0.003 0.043 0.951 -0.087 0.082 
25 0.012 0.045 0.787 -0.077 0.101 
26 0.028 0.047 0.549 -0.064 0.121 
27 0.000 0.044 0.992 -0.085 0.086 
28 0.000 (omitted) 
29 -0.028 0.032 0.374 -0.090 0.034 
30 -0.064 0.036 0.079 -0.135 0.007 
31 0.000 (omitted) 

Intercept 0.063 0.043 0.141 -0.021 0.148 

Var(Int) 1.37E-17  8.08E-17 1.30E-22  1.44H-12 


Notes: Observations = 1,714, Course Sections = 80 


The final outcome investigated was certificate attainment. Table 20 includes the results of the first model 
estimating the effect of the mobile apps on certificate attainment. Students in the treatment group were 
0.3% more likely to earn a certificate compared to the control group, but this difference was not 
statistically significant (p = 0.837). 
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Table 20: Treatment Model of Certificate Attainment for Combined Cohorts, no Interaction 


8 SE p-value 95% Conf. Int. 
Treatment Group 0.003 0.016 0.837 -0.028 0.034 
Credits Attempted Pre-Intervention -0.007 0.001 0.000 -0.009 -0.005 
Credits Earned Pre-Intervention 0.014 0.002 0.000 0.011 0.017 
Race/Ethnicity (White) 
Black -0.039 0.024 0.107 -0.086 0.008 
Other -0.040 0.018 0.029 -0.075 -0.004 
Female (Male) -0.006 0.017 0.739 -0.039 0.028 
Pell Received 0.000 0.000 0.578 0.000 0.000 
Age Category (<18) 
18-21.99 0.023 0.047 0.615 -0.068 0.115 
22-24.99 -0.029 0.053 0.584 -0.132 0.075 
25-29.99 -0.034 0.051 0.507 -0.135 0.067 
30-39.99 0.004 0.051 0.932 -0.095 0.104 
40-49.99 0.058 0.056 0.305 -0.053 0.168 
50+ -0.069 0.069 0.318 -0.203 0.066 
Block ID (Block 1) 
2 -0.058 0.078 0.456 -0.211 0.094 
3 -0.037 0.081 0.649 -0.196 0.122 
4 -0.116 0.084 0.169 -0.281 0.049 
5 -0.027 0.076 0.721 -0.177 0.122 
6 -0.148 0.128 0.248 -0.399 0.103 
7 0.020 0.075 0.790 -0.127 0.167 
8 -0.024 0.072 0.740 -0.165 0.117 
9 0.033 0.075 0.656 -0.113 0.179 
10 -0.047 0.074 0.528 -0.193 0.099 
11 -0.139 0.074 0.060 -0.283 0.006 
12 -0.094 0.069 0.171 -0.229 0.041 
13 0.000 O:135 1.000 -0.265 0.265 
14 -0.043 0.072 0.552 -0.185 0.099 
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15 -0.032 0.071 0.655 -0.171 0.108 
16 -0.010 0.064 0.871 -0.137 0.116 
ibe -0.208 0.113 0.065 -0.430 0,073 
18 -0.114 0.076 0.133 -0.262 0.035 
19 -0.105 0.077 0.170 -0.256 0.045 
20 -0.089 0.073 0.224 -0.232 0.054 
A -0.082 0.078 0.295 -0.236 0.072 
22 -0.086 0.069 0.213 -0.221 0.049 
23 -0.068 0.071 0.342 -0.208 0.072 
24 -0.017 0.072 0.817 -0.158 0.125 
25 -0.093 0.075 0.216 -0.239 0.054 
26 -0.100 0.076 0.189 -0.250 0.049 
Ze -0.058 0.073 0.423 -0.201 0.084 
28 -0.091 0.065 0.162 -0.219 0.037 
29 -0.051 0.066 0.440 -0.179 0.078 
30 -0.113 0.070 0.106 -0.250 0.024 
31 -0.026 0.062 0.678 -0.147 0.096 
Intercept 0.143 0.075 0.057 -0.004 0291 
var (Int) 1.85E-18  4.85E-15 - . 


Notes: Observations = 1,714, Course Sections = 80 

Finally, Table 21 examines whether the effect of the treatment on certificate attainment varied across the 
three course numbers. Students in Math 99 were 3.0% more likely to complete a certificate compared to 
students in the control group, but this difference was not statistically significant (p = 0.350). There were 
also no significant differences between courses in the effect of the treatment, confirmed by the post- 
estimation test (p = 0.432). 


Table 21: Treatment Model of Certificate Attainment for Combined Cohorts, Treatment by Course Number Interaction 


8 SE p-value 95% Conf. Int. 
Treatment Group 0.030 0.032 0.350 -0.033 0.093 
Course (Math 99) 
Engl 99 -0.065 0.069 0.347 -0.199 0.070 
Math 98 -0.017 0.066 0.796 -0.146 0.112 


Treatment by Course 
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Engl 99 -0.051 0.041 0.214 -0.131 0.029 
Math 98 -0.019 0.042 0.649 -0.101 0.063 
Credits Attempted Pre-Intervention -0.007 0.001 0.000 -0.009 -0.005 
Credits Earned Pre-Intervention 0.014 0.002 0.000 0.011 0.017 
Race/Ethnicity (White) 
Black -0.038 0.024 0.115 -0.085 0.009 
Other -0.040 0.018 0.027 -0.076 -0.005 
Female (Male) -0.006 0.017 0.734 -0.039 0.027 
Pell Received 0.000 0.000 0.589 0.000 0.000 
Age Category (<18) 
18-21.99 0.021 0.047 0.648 -0.070 0.113 
22-24.99 -0.031 0.053 G.559 -0.134 0.073 
25-29.99 -0.036 0.051 0.483 -0.137 0.065 
30-39.99 0.004 0.051 0.938 -0.096 0.104 
40-49.99 0.056 0.056 0.323 -0.055 0.166 
50+ -0.068 0.069 0.325 -0.203 0.067 
Block ID (Block 1) 
2 -0.057 0.078 0.460 -0.210 0.095 
3 -0.041 0.081 0.618 -0.200 0.119 
4 -0.118 0.084 0.163 -0.283 0.047 
5 -0.027 0.076 0.721 -0.176 0.122 
6 -0.145 0.128 0.258 -0.396 0.106 
7 0.109 0.058 0.059 -0.004 0.223 
8 0.066 0.054 0.221 -0.040 0.171 
9 0.123 0.057 0.031 0.011 0.235 
10 0.045 0.057 0.432 -0.067 0.157 
11 -0.046 0.056 0.408 -0.156 0.063 
12 -0.007 0.050 0.881 -0.105 0.090 
13 0.088 0.127 0.485 -0.160 0.336 
14 -0.017 0.050 0.739 -0.114 0.081 
15 -0.006 0.048 0.906 -0.100 0.089 
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16 0.016 0.037 0.660 -0.057 0.089 

17 -0.182 0.100 0.069 -0.378 0.014 
18 0.114 0.076 0.132 -0.262 0.034 
19 -0.105 0.077 0.172 0.255 0.046 
20 -0.089 0.073 0.225 0.232 0.055 
21 -0.079 0.079 0.316 0.233 0.075 
9 -0.090 0.069 0.192 0.225 0.045 
23 0.023 0.053 0.666 -0.081 0.127 
24 0.073 0.054 0.180 -0.034 0.180 
25 -0.003 0.057 0.965 0.115 0.110 
26 -0.016 0.060 0.784 0.134 0.101 
27 0.028 0.055 0.607 -0.080 0.137 
28 0.000 (omitted) 
29 -0.023 0.040 0.570 -0.101 0.056 
30 -0.087 0.046 0.059 -0.177 0.003 
31 0.000 (omitted) 

Intercept 0.132 0.076 0.085 -0.018 0.281 

Var(Int) 4.80E-22  2.19E-21 6.25E-26 3.69E-18 


Notes: Observations = 1,714, Course Sections = 80 


Summary of Results and Standardized Effect Sizes 


Given the wide variety of outcomes examined in the impact evaluation and the complexity of the 
statistical models, Table 22 summarizes the results found across all analyses. For each outcome 
investigated, the table includes two key results: the mean difference and the standardized effect size. The 
mean difference is simply the difference between the treatment and control groups on their means for 
the outcomes. While these mean differences are easy to interpret given that they are in the unit of the 
outcome variable, it is difficult to compare the effect of the treatment across outcomes precisely because 
each outcome is in its own units. 


Standardized effect sizes convert these means differences into standardized units. For example, if the 
mean of the GPA outcome was 2.50 and the standard deviation was 0.50, a standardized mean difference 
of one would indicate that the treatment increased GPA by one standard deviation, or 0.50 grade points. 
By standardizing the mean differences, we are better able to compare the size of the treatment effects 
across outcomes. The calculations for computing effect sizes are slightly different if the outcome is 
continuous (e.g. GPA) or dichotomous (e.g. credential attainment). Specifically, a statistic called Hedges’ 
gis used to compute effect sizes for continuous variables and Cohen’s d is used to calculated effect sizes 
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for dichotomous variables. However, these effect sizes are still on roughly the same scale, allowing them 
to be compared to each other. 


A clear patterns emerges when examining the standardized mean differences. The largest effects are 
found for the outcomes specific to the developmental education courses students enrolled in. The effect 
size for passing the developmental education course was 0.131 and the effect on the grade students 
received in their course was 0.092. The effects are more modest, but still important, for the next semester 
outcomes. The effect sizes for next semester outcomes of persistence, GPA, credits earned, and the 
percentage of attempted courses that were passed are all between 0.070-0.093. These results suggest that 
the benefits of being given access to the mobile apps continued into the next semester. 


However, these benefits appeared to wane by the following year when examining students attainment of 
credentials. All of the effect sizes for the credential attainment outcomes were less than 0.05, generally 
considered a small effect. This finding should be interpreted somewhat cautiously given the relatively 
short window for examining students attainment outcomes. For example, Cohort 2 enrolled in 
developmental education courses in Fall 2017, giving them only two years to earn a credential through 
Spring 2019. In fact, it is somewhat surprising that even 8% of students in the sample did in fact earn 
associate’s degrees by Spring 2019 given the fact that the sample all began in developmental education 
and had less than three years to complete an associate’s degree. It may be the case that differences 
between the treatment and control groups on credential attainment will grow larger if we examine 
attainment at a later date. Nevertheless, the results suggest that the intervention had the largest effect on 
students’ short-term outcomes but the effects were more modest on the longer-term outcome of 
credential attainment. The conclusion will discuss the significance of these results in greater detail. 
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Table 22: Standardized Effect Sizes for All Outcomes 


Control Group Treatment Group Combined Sample 
N Mean SD N Mean SD wy pee oe be ae F 

Continuous V ariables 

Dev Ed Grade 1022 1.668 1.486] 988 1.806 1.486] 1.000 0.138 1.486 0.092 

Next Sem GPA 582 1.87 1.262} 598 1.968 1.236 | 0.999 0.098 1.249 0.078 

Next Sem Course Pass Rate 582 0.66 0.377] 598 0.686 0.364 | 0.999 0.026 0.37 0.070 

Next Sem Credit Earned 869 5.153 5.463 | 851 5.616 5.429 | 1.000 0.463 5.446 0.085 
Dichotomous Variables 

Dev Ed Passed 1084 0.513 0.5 | 1038 0.566 0.496 | 1.000 0.053 0.131 

Next Sem Persist 869 0.67 0.471 | 851 0.703 0.457 | 1.000 0.033 0.093 

Credential — Any 869 0.15 0.357} 851 0.155 0.362 | 1.000 0.005 0.026 

Credential — Associate 869 0.078 0.269 | 851 0.083 0.277 | 1.000 0.005 0.042 

Credential — Certificate 869 0.136 0.343 | 851 0.136 0.343 | 1.000 0.000 0.003 
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CONCLUSIONS 


The literature is clear that students required to complete non-credit-bearing developmental education 
coursework are extremely unlikely to persist through college and attain a degree, particularly if they fail to 
complete their developmental education courses. Only 26% of community college students who enroll in 
developmental education courses and pass all of the courses they attempt earn an associate’s degree or 
certificate within six years, and their completion rate drops to 12% if they do not pass their 
developmental education courses (Chen & Simone, 2016). At BPCC, only 14% of students who enrolled 
in a developmental education course during Fall 2012 earned a credential by Spring 2016 (Giant 
Consulting & Evaluation, 2016). Challenges this great, and students facing such challenges, require bold 
and innovative solutions. 


There is a risk of encountering obstacles anytime an attempt is made to change the status quo. This is 
particularly true when proposed innovations rely on new technologies to transform practices and 
processes. It is not surprising that BPCC encountered various challenges while implementing mobile 
apps for developmental education courses and the IDP, including technical difficulties, confusion among 
stakeholders over the nature of the initiative, and reluctance to abandon established practices in favor of 
innovative approaches. Despite these challenges, BPCC was able successfully implement all of the 
strategies and activities included in the logic model in its FITW application while maintaining buy-in for 
the initiative among stakeholders, in particular the faculty and front-line staff who are tasked with 
incorporating these new technologies into their work. Confusion and uncertainty continue to be resolved 
as faculty and staff become more knowledgeable of the purpose of these approaches and how to 
effectively utilize these strategies in their practice. 


The results of this evaluation provide evidence that the strategies BPCC is employing may be effective at 
improving outcomes for one of the most disadvantaged student populations in higher education, namely 
college students unprepared for the rigors of college-level coursework. Although there is an opportunity 
to increase faculty and student engagement with the mobile apps moving forward, evidence suggests that 
students given access to the apps outperform their peers in the same developmental education courses. 
Students in the treatment group were significantly more likely to pass their developmental education (d= 
0.131) courses and earned significantly better grades (g = 0.092) compared to students in the control 
group. Even mote, these benefits appear to carry over, as students given access to the mobile apps were 
more likely to persist to the next semester (d = 0.093) and had better course performance than their peers 
in the subsequent semester. However, it appears that this effect began to wane over time, as no 
significant differences were found between the treatment and control groups on the outcomes related to 
credential attainment, and all standardized effect sizes were less than 0.05, generally considered to be a 
small effect. 


It is important to properly contextualized these results. One way to do so is to compare these findings to 
other rigorous evaluations of educational interventions. The Investing in Innovation (13) program was 
established by the American Recovery and Reinvestment Act of 2009 (ARRA) and administered by the 
US Department of Education Office of Innovation and Improvement. Sixty-seven grants were awarded 
through the i3 program, for a total of $679 million disbursed to 13 grantees. These grants included a 
required third-party impact evaluation that used rigorous methods to estimate the effect of the 
intervention on student outcomes, such as a randomized controlled trial. Overall, only 18% of the impact 
evaluations of 13 grants found a statistically significant positive impact on at least one student academic 
outcome, and only 8% of new interventions supported by 13 Development grants significantly improved 
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at least one academic outcome (Boulay et al., 2018). Similarly, only three of the 23 evaluations that 
examined mathematics achievement as an outcome found a statistically significant positive effect, and 
two of the 23 evaluations found negative effects. 


These findings from the 13 program make the results from the present evaluation all the more impressive. 
Although creating the Open Campus mobile apps required an up-front investment of technology and 
training, the intervention itself is relattvely low-cost, sustainable, and scalable. Even so, the impact 
evaluation found moderate effects on a variety of students’ educational outcomes, including their 
persistence and academic performance in the next semester in college. This is all the more impressive 
given that the mobile apps were not designed to be used in subsequent college courses, suggesting that 
the effects on next semester outcomes were due to the knowledge students had gained and retained from 
using the mobile apps. Given the significant expense associated with some of the most impactful 
interventions aimed at improving the outcomes of developmental education students, such as the City 
University of New York’s ASAP program, the results from the present evaluation are promising for 
other postsecondary institutions seeking to identify sustainable interventions effective at improving the 
outcomes of students requiring remediation. 


Perhaps the greatest barrier to implementation of the mobile apps is that they were not true mobile apps 
at the time the intervention occurred. BPCC has now released fully functioning Open Campus mobile 
apps that are downloadable through the Apple App Store and Google Play. The results from the fourth 
year site visit suggest that students and faculty have mote positive perceptions of this version of the 
intervention compared to the previous version. It appears likely that this version of the intervention will 
be easier to implement by faculty and easier to use by students, both of which could improve the efficacy 
of the intervention. Although there are currently no plans to rigorously evaluate this version of the 
intervention on student outcomes, it will be important for future research to examine whether the fully 
functioning mobile apps and similar technological innovations can continue to drive improvements in 
students’ college outcomes. 
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